{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 分类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 加载数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_openml\n",
    "mnist_data = fetch_openml(\"mnist_784\")\n",
    "mnist_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "X,y =mnist_data[\"data\"],mnist_data[\"target\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(70000, 784)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(70000,)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAOcAAADnCAYAAADl9EEgAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAFvklEQVR4nO3doWvUfxzH8bsfcyiCToOCXLLsLxAVDDYVDGqQJYMMBLELChpcFpthIpiWDEtmF8SiRVFkKAjDsCBom2Um+THY9/3VO+e97u7xiL74bl/EJ1/ww32vu7m52QHy/DfsGwC2J04IJU4IJU4IJU4INdWy+69c2Hnd7f7QkxNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCiRNCTQ37Bvgz6+vrA+07adeuXeU+Ozv7j+5kPHhyQihxQihxQihxQihxQihxQihxQijnnEPw+vXrxu3+/fvltS9evCj3z58/93VPv2xubjZu3W63vHZ6errcT5482dc9dTr1fXU6nc6FCxfK/ciRI+Xedm+9Xq/cd4InJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4Tqtpwf1YdL9OX48eON25s3b8pr9+/fX+5nzpwp96NHj5b7gQMHGrc9e/aU13758qXc27x9+7ZxW1lZKa9tO4Nt03aOOjc317g9ePBgoN/d6XS2vXlPTgglTgglTgglTgglTgglTgglTgg1kZ/nbHu367t37wb6+ceOHSv3EydONG43b94sr7106VJf90Tt5cuX5b62tvaP7uR/npwQSpwQSpwQSpwQSpwQSpwQSpwQamw/z7mwsNC4LS4ultcOeqa1tLRU7pcvXx7o5zN2fJ4TRok4IZQ4IZQ4IZQ4IZQ4IdTYHqVUr0pse43ioUOHyv3Vq1fl3vYKyYMHD5Y7E8dRCowScUIocUIocUIocUIocUIocUKosX015sWLFxu35eXl8tqNjY1yX11dLffTp0+XO/wOT04IJU4IJU4IJU4IJU4IJU4IJU4INbbnnLOzs43b9PR0ee3379/L/dy5c+V+7dq1cr9161bjNjMzU17bdu+MD09OCCVOCCVOCCVOCCVOCCVOCCVOCDW2762tPH/+vNyfPXtW7k+ePCn39fX1cq/+zufn58trr1y5Uu6nTp0qdyJ5by2MEnFCKHFCKHFCKHFCKHFCKHFCqIk85xzU2tpauS8uLpb7vXv3Gre27w6dmqo/gnvjxo1yv337drn77tChcM4Jo0ScEEqcEEqcEEqcEEqcEMpRyhB8/fq1cVtYWCivbfs424cPH/q6p1+WlpYat/Pnz5fX7t27d6DfPcEcpcAoESeEEieEEieEEieEEieEEieEcs45Yn78+FHuT58+Lffr16+X+7dv3xq3R48elddevXq13GnknBNGiTghlDghlDghlDghlDghlDghlHPOCbOyslLuZ8+ebdxa/q10VldXy73X65X7BHPOCaNEnBBKnBBKnBBKnBBKnBBKnBDKOSdbVF8h+PDhw/Lau3fvlvudO3f6uqcJ4JwTRok4IZQ4IZQ4IZQ4IZQ4IdTUsG+ALHNzc41b21EKf5cnJ4QSJ4QSJ4QSJ4QSJ4QSJ4QSJ4RyzskWMzMzjVvbqzHbdv6MJyeEEieEEieEEieEEieEEieEEieEcs7JFp8+fWrcut1t3+D42zt/xpMTQokTQokTQokTQokTQokTQokTQjnnZIvl5eW+r929e/dfvBM8OSGUOCGUOCGUOCGUOCGUOCGUOCFUt+Vdo15EOmYeP35c7vPz843b4cOHy2vfv39f7tU7cSfcth+E9eSEUOKEUOKEUOKEUOKEUOKEUI5SJszHjx/LfWNjo3Hbt29feW2v1+vrnnCUAiNFnBBKnBBKnBBKnBBKnBBKnBDKOScMn3NOGCXihFDihFDihFDihFDihFDihFBtXwG47fkLsPM8OSGUOCGUOCGUOCGUOCGUOCHUT48O4rmnsp2iAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "some_digit = X[35500]\n",
    "some_digit_image = some_digit.reshape(28,28)\n",
    "plt.imshow(some_digit_image, cmap = matplotlib.cm.binary, interpolation = \"nearest\")\n",
    "plt.axis(\"off\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'5'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y[35500]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "让我们打乱训练集。这可以保证交叉验证的每一折都是相似（你不会期待某一折缺少某类数字）。而且，一些学习算法对训练样例的顺序敏感，当它们在一行当中得到许多相似的样例，这些算法将会表现得非常差。打乱数据集将保证这种情况不会发生"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "TRAIN_SIZE = 60000\n",
    "\n",
    "X_train,X_test,y_train,y_test = X[:TRAIN_SIZE],X[TRAIN_SIZE:],y[:TRAIN_SIZE],y[TRAIN_SIZE:]\n",
    "shuffle_index = np.random.permutation(60000)\n",
    "X_train, y_train = X_train[shuffle_index], y_train[shuffle_index]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练二分类器"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在我们简化一下问题，只尝试去识别一个数字，比如说，数字 5。这个“数字 5 检测器”就是一个二分类器，能够识别两类别，“是 5”和“非 5”。让我们为这个分类任务创建目标向量："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_5 = (y_train == '5') # True for all 5s, False for all other digits.\n",
    "y_test_5 = (y_test == '5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在让我们挑选一个分类器去训练它。用随机梯度下降分类器 SGD，是一个不错的开始。使用 Scikit-Learn 的SGDClassifier类。这个分类器有一个好处是能够高效地处理非常大的数据集。这部分原因在于 SGD 一次只处理一条数据，这也使得 SGD 适合在线学习（online learning）。我们在稍后会看到它。让我们创建一个SGDClassifier和在整个数据集上训练它。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGDClassifier(random_state=42)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import SGDClassifier\n",
    "sgd_clf = SGDClassifier(random_state=42)\n",
    "sgd_clf.fit(X_train, y_train_5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sgd_clf.predict([some_digit])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "分类器猜测这个数字代表 5（True）。看起来在这个例子当中，它猜对了。现在让我们评估这个模型的性能。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 对性能的评估"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "评估一个分类器，通常比评估一个回归器更加玄学。所以我们将会花大量的篇幅在这个话题上。有许多量度性能的方法，所以拿来一杯咖啡和准备学习许多新概念和首字母缩略词吧。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用交叉验证测量准确性"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在交叉验证过程中，有时候你会需要更多的控制权，相较于函数cross_val_score()或者其他相似函数所提供的功能。这种情况下，你可以实现你自己版本的交叉验证。事实上它相当简单。以下代码粗略地做了和cross_val_score()相同的事情，并且输出相同的结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\dongc\\.conda\\envs\\MachineLearning\\lib\\site-packages\\sklearn\\model_selection\\_split.py:297: FutureWarning: Setting a random_state has no effect since shuffle is False. This will raise an error in 0.24. You should leave random_state to its default (None), or set shuffle=True.\n",
      "  FutureWarning\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8922\n",
      "0.95995\n",
      "0.96245\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.base import clone\n",
    "\n",
    "skfolds = StratifiedKFold(n_splits =3, random_state = 42)\n",
    "for train_index, test_index in skfolds.split(X_train, y_train_5):\n",
    "    clone_clf = clone(sgd_clf)\n",
    "    X_train_folds = X_train[train_index]\n",
    "    y_train_folds = (y_train_5[train_index]) # \n",
    "    X_test_fold = X_train[test_index]\n",
    "    y_test_fold = (y_train_5[test_index])\n",
    "    clone_clf.fit(X_train_folds, y_train_folds)\n",
    "    y_pred = clone_clf.predict(X_test_fold)\n",
    "    n_correct = sum(y_pred == y_test_fold)\n",
    "    print(n_correct/len(y_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们使用cross_val_score()函数来评估SGDClassifier模型，同时使用 K 折交叉验证，此处让k=3。记住：K 折交叉验证意味着把训练集分成 K 折（此处 3 折），然后使用一个模型对其中一折进行预测，对其他折进行训练。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.8922 , 0.95995, 0.96245])"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "cross_val_score(sgd_clf, X_train, y_train_5, cv=3, scoring = \"accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "哇！在交叉验证上有大于 95% 的精度（accuracy）？这看起来很令人吃惊。先别高兴，让我们来看一个非常笨的分类器去分类，看看其在“非 5”这个类上的表现。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.base import BaseEstimator\n",
    "class Never5Classifier(BaseEstimator):\n",
    "    def fit(self, X, y = None):\n",
    "        pass\n",
    "    def predict(self, X):\n",
    "        return np.zeros((len(X),1), dtype = bool)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.90965, 0.90995, 0.90935])"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "never_5_clf = Never5Classifier()\n",
    "cross_val_score(never_5_clf, X_train, y_train_5, cv=3, scoring=\"accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "没错，这个笨的分类器也有 90% 的精度。这是因为只有 10% 的图片是数字 5，所以你总是猜测某张图片不是 5，你也会有 90%的可能性是对的。\n",
    "\n",
    "这证明了为什么精度通常来说不是一个好的性能度量指标，特别是当你处理有偏差的数据集，比方说其中一些类比其他类频繁得多。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 混淆矩阵"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对分类器来说，一个好得多的性能评估指标是混淆矩阵。大体思路是：输出类别 A 被分类成类别 B 的次数。举个例子，为了知道分类器将 5 误分为 3 的次数，你需要查看混淆矩阵的第五行第三列。\n",
    "\n",
    "为了计算混淆矩阵，首先你需要有一系列的预测值，这样才能将预测值与真实值做比较。你或许想在测试集上做预测。但是我们现在先不碰它。（记住，只有当你处于项目的尾声，当你准备上线一个分类器的时候，你才应该使用测试集）。相反，你应该使用cross_val_predict()函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import cross_val_predict\n",
    "y_train_pred = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True, False, False, ..., False, False, False])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "就像 cross_val_score()，cross_val_predict()也使用 K 折交叉验证。它不是返回一个评估分数，而是返回基于每一个测试折做出的一个预测值。这意味着，对于每一个训练集的样例，你得到一个干净的预测（“干净”是说一个模型在训练过程当中没有用到测试集的数据）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在使用 confusion_matrix()函数，你将会得到一个混淆矩阵。传递目标类(y_train_5)和预测类（y_train_pred）给它。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[51998,  2581],\n",
       "       [ 1127,  4294]], dtype=int64)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "confusion_matrix(y_train_5, y_train_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "混淆矩阵中的每一行表示一个实际的类, 而每一列表示一个预测的类。该矩阵的第一行认为“非 5”（反例）中的 51998 张被正确归类为 “非 5”（他们被称为真反例，true negatives）, 而其余 2581 被错误归类为\"是 5\" （假正例，false positives）。第二行认为“是 5” （正例）中的 1127 被错误地归类为“非 5”（假反例，false negatives），其余 4294 正确分类为 “是 5”类（真正例，true positives）。一个完美的分类器将只有真反例和真正例，所以混淆矩阵的非零值仅在其主对角线（左上至右下）。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准确率与召回率"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6245818181818181"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import precision_score, recall_score\n",
    "precision_score(y_train_5, y_train_pred)# 4294/(4294+2581)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7921047777162885"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recall_score(y_train_5, y_train_pred) # == 4294/ (4294 + 1127)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 准确率/召回率之间的折衷"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6984385165907612"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "f1_score(y_train_5, y_train_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scikit-Learn 不让你直接设置阈值，但是它给你提供了设置决策分数的方法，这个决策分数可以用来产生预测。它不是调用分类器的predict()方法，而是调用decision_function()方法。这个方法返回每一个样例的分数值，然后基于这个分数值，使用你想要的任何阈值做出预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([4869.85004626])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_scores = sgd_clf.decision_function([some_digit])\n",
    "y_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ True])"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "threshold = 0\n",
    "y_some_digit_pred = (y_scores > threshold)\n",
    "y_some_digit_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SGDClassifier用了一个等于 0 的阈值，所以前面的代码返回了跟predict()方法一样的结果（都返回了true）。让我们提高这个阈值："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([False])"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "threshold = 200000\n",
    "y_some_digit_pred = (y_scores > threshold)\n",
    "y_some_digit_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这证明了提高阈值会降调召回率。这个图片实际就是数字 5，当阈值等于 0 的时候，分类器可以探测到这是一个 5，当阈值提高到 20000 的时候，分类器将不能探测到这是数字 5。\n",
    "\n",
    "那么，你应该如何使用哪个阈值呢？首先，你需要再次使用cross_val_predict()得到每一个样例的分数值，但是这一次指定返回一个决策分数，而不是预测值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_scores = cross_val_predict(sgd_clf, X_train, y_train_5, cv=3, \n",
    "                            method=\"decision_function\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在有了这些分数值。对于任何可能的阈值，使用precision_recall_curve(),你都可以计算准确率和召回率:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_recall_curve\n",
    "precisions, recalls, thresholds = precision_recall_curve(y_train_5, y_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最后，你可以使用 Matplotlib 画出准确率和召回率，这里把准确率和召回率当作是阈值的一个函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEKCAYAAAACS67iAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAy3UlEQVR4nO3dd3xUVdrA8d+TISEEkhCQEhJKQEDpEnqRXq24otixseii6/q6q66ru+ra1oJdZFdUbKigYAFFQJAiXVpoRmoA6QmEFjI57x9nQkIIYQIzuVOer5/53Jl7bnnmEp+cnHvuOWKMQSmlVPCLcDoApZRSvqEJXSmlQoQmdKWUChGa0JVSKkRoQldKqRChCV0ppULEGRO6iIwRkV0isuo05SIir4pIuoisEJHWvg9TKaXUmXhTQ38P6F9C+QCgoec1DHjr3MNSSilVWmdM6MaYn4B9JWxyBTDWWPOByiKS6KsAlVJKeaecD46RBGwt9DnDs25H0Q1FZBi2Fk/FihVTL7jgglKfbO/hvWzK3HRWgapSEhCk0Ec5dZ0ILnERIRG4IlwnykSkYJ9CnyMkomBdkePlfwaIjIikvKs8IoIgJ45f+DwREnGiXKmSuN2wbNmp62vVgsREOH4cVqw4tTw5GWrUgKNHIS3t1PK6deG88+DQIVi79tTylBSoUgUOHoT16+26ihXhLFLfCUuWLNljjKlWXJkvEnpx/zcVO56AMWY0MBqgTZs2ZvHixWd1wowDGRw+fjj/mEVOXPDZ2zJTJNyzKfNFHL6K8fDxw+S4c8jNyyU3L5fj7uMczDlInsnDGEOeybPvKfS+yPqjuUfJceecWHe615HjR8g+nk12jn3lr3fnuU/azm3cuPPcHD5+GLdxn3a7/G2zc7LxRpQrimbVm1G9YnVqVqpJQnQCsVGx1KhUg/Ku8kS5oogtH0tipUTiyscR5YqifLnyVI6uTExkDBGi/QJCkdsNERHgqUeQlwfbt5+6XVycfbndsOOUKijEx0NsrE34O3eeWl65MlSqBDk5sGvXqeVVqkBMjP2FsGePXRcVBdWrn/VXQ0Q2n67MFwk9A6hd6HMyUMyl853kuGR/Hl4FgL2H97Lz0E5y3DnkuHPIzsnm4LGDZB3L4lDOIY7nHSfHncPG/RvZmLmRnYd2smLnCrKOZnHo+CGvz1M5ujIVIyuSUCGB5LhkkmOTqR1fm/oJ9WlctTH1E+qTUCFBE7+fHT8OS5dChQo2wdard27H+/BDePhhuPhiiI6G5cvhm28gKan47V0uWxs/ncjIksujokouj44uudxXfJHQvwJGiMg4oD2QZYwp5nedUt6rGlOVqjFVz2rfI8ePcODYgRO/DPYf3c/2g9s5cvwIOe4cjuQeIfNoJtk52ew/sp/Dxw+z+/BuNuzfwC87fmHnoZOrYhESQfWK1UmOS6Z9UnuSYpNoXqM59SrXo17lelSKquSLrxxWjIGnnrK16BtvtE0XhX37LQwcaGvOLhf8+9/w0UeweLFtsjDGNpE0bQrlimSxxYuhZk3o2RN++KGg5pybWzbfzUlnTOgi8gnQHThPRDKAfwKRAMaYUcBkYCCQDhwGbvVXsEp5o0JkBSpEVjjr/Y8cP8KmzE2s3bOWTZmb2HN4D7sO7WLd3nW8veRtcvMKMoMgtKjRgiHNhtC9XnfaJ7U/cb9AFe/33227db5LL7UJ+Pff4e67bfNF16627IUX4KGHCratWNEue/eGGTPs52nToEMHeP11aN0aOneGESNsLd3thrlzYeVKqFOn7L6jU86Y0I0x152h3AB/8kUwx48fJyMjg6NHj/ricGEnOjqa5ORkIiMjnQ4lqFWIrMCF1S7kwmoXnlJmjOFgzkFW7lzJ1gNbSduVxpT0KTw8/WEAqlesTs+UnrRPas/gJoNJijvN3/hh5sUXYeZMGDPm5GQ+axa0aFF8+zXYmnvFivam4xtv2HXG2CYUsOs7doT//Mcm/rw8u75pU7t0uWyzy8UX++VrBRxxajz04m6Kbty4kdjYWKpWraq1nFIyxrB3714OHjxISkqK0+GEnd2HdvNZ2mfM3jKbOVvmsO3gNgAaVmnIjS1u5NJGl9KwSkNiy8c6HGnZuvVWeO+9gs933GHbyF96CfbvtzcVz8SYgpub+fLyICPDJvsJE+C222DdOhg8GNasga+/tjX/UCQiS4wxbYotC6SEvmbNGi644AJN5mfJGMPatWu58MJTa5aqbK3ds5YJqyfw9fqvWbBtwYn1lzS8hGubXsu1za4lyhXlYIT+k5dn27z79YO334Z337XrZ8+GLl38f/7sbNvzJFQFVULXZHRu9BoGnk2Zm/h568/Mz5jPp2mfsvPQTipFVWJwk8Hc0vIWutXr5nSIPnP4cEE795VXwjPP2L7ecXGOhhVSSkro2hdLKT+rV7ke1zW/jlcGvML2/9vOlBumcOUFVzJ+9Xi6v9+dgR8NJG1XMU+tBJmcnIJkDvCXv9gHaDSZlx1N6EW4XC5atWpFs2bNGDx4MIcPHz7nYz722GNMmzbttOWjRo1i7Nix53weFfgiJIL+5/fng0Ef8PsDv/Nwl4eZvnE6qaNTeWDqA/ye/bvTIZ61Tz6xy2rVbLt3uNyIDCTa5FJEpUqVyM62TynecMMNpKamcv/9958od7vduFwup8I7o0C4hqp0th3Yxv1T7+eztM+oHF2ZMZePYdCFg5wOq1TmzrUP7axdC3372v7lyj+0yeUsde3alfT0dGbOnEmPHj24/vrrad68OW63m7/+9a+0bduWFi1a8Pbbb5/Y5z//+Q/NmzenZcuWPOTpQDt06FDGjx8PwEMPPUSTJk1o0aIFDzzwAAD/+te/eOGFFwBYtmwZHTp0oEWLFgwaNIj9+/cD0L17dx588EHatWtHo0aNmD17dlleCuVHSXFJfHr1p8y7bR4J0Qlc9dlV3Pzlzew7UtKYeIHjwAF7s7NXL+jfX5O5k3zxpKjfdO9+6rprrrEPHxw+bJ8kK2roUPvasweuvvrkspkzvT93bm4uU6ZMoX9/O3LwwoULWbVqFSkpKYwePZr4+HgWLVrEsWPH6Ny5M3379mXt2rVMnDiRBQsWEBMTw759J/8PuW/fPr788kvWrl2LiJCZmXnKeW+++WZee+01unXrxmOPPcbjjz/Oyy+/fCKmhQsXMnnyZB5//PESm3FU8OlYuyOr/7Sav0//OyPnj2Tqb1P56rqvaJfUzunQShQfb5c33+xsHEpr6Kc4cuQIrVq1ok2bNtSpU4fbb78dgHbt2p3o3z116lTGjh1Lq1ataN++PXv37uXXX39l2rRp3HrrrcTExABQpUqVk44dFxdHdHQ0d9xxB1988cWJ7fJlZWWRmZlJt26218Mtt9zCTz/9dKL8qquuAiA1NZVNmzb55fsrZ0WXi+alfi8x9cap5Jk8Oo/pzDfrv3E6rNMq3MP4sceci0NZAV1DL6lGHRNTcvl555WuRp6vQoUKLCtmnM2KhW7fG2N47bXX6Nev30nbfPfddyX2oS9XrhwLFy5k+vTpjBs3jtdff50ZM2Z4HVv58uUBe+M2NxwGpghjfRr04Zc//kL397tz+SeX83CXh3my55MBNUiY2w033GDHWNm//9SHf1TZC5yfjiDSr18/3nrrLY4fPw7A+vXrOXToEH379mXMmDEnesYUbXLJzs4mKyuLgQMH8vLLL5/yiyM+Pp6EhIQT7eMffPDBidq6Cj9JcUksvnMxV15wJU/PeZpHpj9yynDLTklPh+bN4Z13bGL35olP5X8BXUMPVHfccQebNm2idevWGGOoVq0aEydOpH///ixbtow2bdoQFRXFwIEDefrpp0/sd/DgQa644gqOHj2KMYaRI0eecuz333+f4cOHc/jwYerXr8+7+Y/ZqbAUHx3P+GvGM3TiUJ6d+ywAz/R+xtGY8vKgYUP7ftcuqF275O1V2dFuiyFGr2Focue5uXXSrXyw4gOe7vk0D3d92LFYEhIgM9MOnKW3cspeSd0WtYauVBBwRbgYdeko9h3Zx99n/J0IieDBLg+WeRzLltlkDsVPuaacpW3oSgWJmMgYvrz2SzrX7szD0x9mxkbvb6j7Sv36dvnf/9pZeFRg0YSuVBCJdEUy5YYppCSkcM3n17A587TTS/rcjh3w00/2sf477iiz06pS0ISuVJCJLR/LhGsmkJ2TzfVfXM9x9/EyOW+tWnDZZXZ4WhWYNKErFYRa1WzF6MtGM2/rPD5e+bHfz7egYEj3kB5rPNhpQlcqSF3f/HrqxtflwWkPcuDYAb+eq0MHu9xcdi086ixoQi+i8PC5l112WbHjrZyLevXqsWfPHsCO7KjU2SoXUY7Rl41m56GdDPt6GHkmzy/n2bix4H04TLQczDShF5H/6P+qVauoUqUKb+TPTKtUAOrboC+PdH2ET9M+Zfzq8X45R1ISTJyotfNgoAm9BB07dmTbNjvZ72+//Ub//v1JTU2la9eurPV0wt25cyeDBg2iZcuWtGzZknnz5gFw5ZVXkpqaStOmTRk9erRj30GFvse7P05yXDIv/fySz4+dmWlvhHbtqrXzYBCwDxbd9919LPt9mU+P2apmK17u/7JX27rdbqZPn35itMVhw4YxatQoGjZsyIIFC7j77ruZMWMG9957L926dePLL7/E7XafmBxjzJgxVKlShSNHjtC2bVv+8Ic/ULVqVZ9+H6XAPnT0QMcHuO/7+/hx44/0SOnhs2MnJ8OhQ3agO89gnyqAaQ29iPzhc6tWrcq+ffvo06cP2dnZzJs3j8GDB9OqVSv++Mc/smPHDgBmzJjBXXfdBdj293jP4NCvvvoqLVu2pEOHDmzdupVff/3Vse+kQt+tF91KzUo1uWfKPeTm+W4kzkOH7HJQcE2gFLYCtobubU3a1/Lb0LOysrj00kt54403GDp0KJUrVy52WN3izJw5k2nTpvHzzz8TExND9+7dOXr0qH8DV2Etrnwcz/d5npu+vIkJqydwbbNrz/mYngm1uOwyHRo3WGgN/TTi4+N59dVXeeGFF6hQoQIpKSl8/vnngB0Pffny5QD06tWLt956C7DNNAcOHCArK4uEhARiYmJYu3Yt8+fPd+x7qPBxTdNrqF6xOq8seMUntfQXX7TLYgYFVQFKE3oJLrroIlq2bMm4ceP46KOPeOedd2jZsiVNmzZl0qRJALzyyiv8+OOPNG/enNTUVNLS0ujfvz+5ubm0aNGCRx99lA75nXiV8qMoVxRP93yanzN+ZsLqCT47boMGPjuU8jMdPjfE6DUMb+48NymvpFAnvg6zb51d4gxaJTEGjhyxEz7rIFyBpaThc7WGrlQIcUW4uKfdPczdOpcp6VPO+jiXXAJNmmgyDzaa0JUKMfe2v5ek2CSem/vcWe1vDEyZog8SBaOAS+iBMmdiMNJrpwDKlyvPPe3u4afNP5G+L73U+0+caJcPPeTbuJT/BVRCj46OZu/evZqYzoIxhr179xKtfyMrONFt8Z2l75R63/fft8s77/RlRKosBFQ/9OTkZDIyMti9e7fToQSl6OhokpOTnQ5DBYB6lesxuMlgRs4fyYh2I0iKS/Jqv7Q0mDQJhg4tmJ1IBY+ASuiRkZGkpKQ4HYZSIeHfPf/N56s/Z+T8kbzQ9wWv9qlZE665Bv7xDz8Hp/wioJpclFK+06hqI/5w4R94Y9EbZB7N9GqfqlXh00+173mw8iqhi0h/EVknIukicsqtEhGJF5GvRWS5iKSJyK2+D1UpVVp/6/w3juYe5Ys1X5xx29xc+Oc/YeXKMghM+cUZE7qIuIA3gAFAE+A6EWlSZLM/AauNMS2B7sCLIhLl41iVUqXUtlZbGiQ04N1l755x2zZt4Ikn4LffyiAw5Rfe1NDbAenGmA3GmBxgHHBFkW0MECv2sbRKwD7Ad0O+KaXOiogwLHUYc7bMYfXu1SVu6xmeiAEDyiAw5RfeJPQkYGuhzxmedYW9DlwIbAdWAn825tT5sERkmIgsFpHF2pNFqbJxU4ubiHJF8eaiN0+7zbFjBe/Lly+DoJRfeJPQixsMomhH8X7AMqAW0Ap4XUTiTtnJmNHGmDbGmDbVqlUrZahKqbORGJvI4CaDGbt8LEeOHyl2m7Q0u8zvg66CkzcJPQOoXehzMrYmXtitwBfGSgc2Ahf4JkSl1Lka0mwIB3MOMnvL7GLLL7oI5syBG28s48CUT3mT0BcBDUUkxXOjcwjwVZFttgC9AESkBtAY2ODLQJVSZ69XSi9c4uKH334otjwnBzp3tqMrquB1xn8+Y0wuMAL4HlgDfGaMSROR4SIy3LPZk0AnEVkJTAceNMbs8VfQSqnSqRBZgb4N+vL+8vc57j5+UllOjn2g6PnnHQpO+YxXT4oaYyYDk4usG1Xo/Xagr29DU0r50h9T/8iU9ClMSZ/C5Y0vP7F+0iTIzAR9SDv46R9YSoWJgQ0HElc+jq/Wndximj9DYo8eDgSlfEoTulJhItIVSa+UXkxcO5HsnOwT6+fOhU6d7GP/KrhpQlcqjIxoN4K9R/Yy+VfbgpqVBYsWQc+eDgemfEITulJhpFvdblSpUIVJ6+wk55GRMHYsXHedw4Epnwio4XOVUv7linBxWaPL+GrdVxhjiIkRbrjB6aiUr2gNXakw071ed/Yf3c8vv//Chx/qYFyhRBO6UmHmkoaXECERvLvwc266yTa5qNCgCV2pMFOtYjX6NejHJ2kfANCihcMBKZ/RhK5UGOp/fn/2Ht8GCb/RVx8JDBma0JUKQ5c0vMS+afw1sbHOxqJ8RxO6UmGoQZUGRO5vSpXOZ56aTgUPTehKhalHrhzMvtjZ7Mze6XQoykc0oSsVpvqd3w+A6RunOxyJ8hVN6EqFoeeeg+f/kkqlqErM3lz8pBcq+GhCVyoMvfgirF4VSY96Pfj+t++dDkf5iCZ0pcLM0aOwezcYA73r92Zj5kY27NcJxkKBJnSlwszcuXZ5yy12ajqAWZtmORiR8hVN6EqFmZ9+snOH3n03XFjtQmpWqsk3v37jdFjKBzShKxVm2rWDJ56A+HiIkAh61+/NT5t/Is/kOR2aOkea0JUKM5dcAo88UvC5R70e7Dm8h7Rdac4FpXxCE7pSYWTLFpgyxd4Yzde3gR3Mpehcoyr4aEJXKoxMmAADB8L+/QXrkuOS6VS7E5+s+sS5wJRPaEJXKowsXAh16kBi4snrr216LWm701ize40zgSmf0ISuVBhZvBjatDl1/VUXXgXA1N+mlnFEypc0oSsVJg4csNPNtWp1allyXDK1YmuxaPuiMo9L+Y4mdKXCxC+/QGwstG1bfHn7pPbM3Tq3bINSPqUJXakw0awZ9O0LnToVX961Tlc2ZW5ic+bmsg1M+YwmdKXCRNWq8PnnEBdXfPnAhgMB+Ga9PjUarDShKxUm/vY32wf9dBqf15iUyin8sOGHsgtK+ZQmdKXCwIED8PzzsHRpydv1rt+bHzf9iDvPXTaBKZ/ShK5UGFixwi6L6+FSWM+Unhw4dkB7uwQpTehKhYGXX7bLMyX0/uf3JzIikklrJ/k7JOUHmtCVCgO7d9tlUlLJ21WOrkzbpLbajh6kNKErFQaqVYMbb/Ru2z71+7B0x1Kyjmb5Nyjlc14ldBHpLyLrRCRdRB46zTbdRWSZiKSJiE5/olQAGT8exo71btsudbpgMPyc8bN/g1I+d8aELiIu4A1gANAEuE5EmhTZpjLwJnC5MaYpMNj3oSqlzobbDQcPgoh323eq3YnyrvJ8l/6dfwNTPudNDb0dkG6M2WCMyQHGAVcU2eZ64AtjzBYAY8wu34aplDpb115rHyZye9kTMSYyhl71e/Htr9/6NzDlc94k9CRga6HPGZ51hTUCEkRkpogsEZGbizuQiAwTkcUisnh3/l0apZRfbfY8ye9yeb9P1zpdSd+Xzt7De/0TlPILbxJ6cX+omSKfywGpwCVAP+BREWl0yk7GjDbGtDHGtKlWrVqpg1VKlY4xdsjc7t1Lt1/H5I4A2o4eZLxJ6BlA7UKfk4HtxWzznTHmkDFmD/AT0NI3ISqlztaqVXbZ6JTqVcnaJrWlXEQ55myZ4/uglN94k9AXAQ1FJEVEooAhQNHJBycBXUWknIjEAO0BnfpEKYfNm2eXt91Wuv1iImNoU6sNMzbO8H1Qym/OmNCNMbnACOB7bJL+zBiTJiLDRWS4Z5s1wHfACmAh8D9jzCr/ha2U8kaHDvDPf0Jqaun37ZXSi6U7lnLk+BHfB6b8opw3GxljJgOTi6wbVeTz88DzvgtNKXWuWra0r7PRplYb3MbNkh1L6FKni28DU36hT4oqFcLmzbMjLZ6Ni+teTIREMOXXEsbcVQFFE7pSIWrrVujcGd577+z2r1KhCl3qdGFKuib0YKEJXakQ9csvdnm6OUS9cXGdi1mxcwUHjx30TVDKrzShKxWi8sdAb9r07I/RIbkDbuNm+c7lvglK+ZUmdKVC1IIF0Ljx6ecQ9UbrxNYALN6+2EdRKX/ShK5UCDLGJvQOHc7tOImxiSTHJbNw20LfBKb8yqtui0qp4PP11xATc+7HaZfUTqekCxJaQ1cqBIlA+/bQvPm5H6t9UnvS96Wz65AOohroNKErFYLGjIE33vDNsbrW6Qqg47oEAU3oSoWg22+HESN8c6zUWqlUKFeB2Ztn++aAym+0DV2pEGMMlCsHg300b1iUK4r2ye2ZvUUTeqDTGrpSIWb7dsjNtU+J+kq3ut1YumMpmUczfXdQ5XOa0JUKMTNn2mWLFr475sV1L8ZgmLd1nu8OqnxOE7pSIeY7z9zOvujhkq9Dcged8CIIaEJXKsR88AFkZkLlyr47ZkxkDKmJqZrQA5wmdKVCUHy874/ZqXYnFm1fRI47x/cHVz6hCV2pELJunX2oaJ4fmro71e7E0dyjLP9dB+oKVJrQlQohsz09CyMjfX/sjskdAZi7da7vD658QhO6UiFk5Uo7fkvr1r4/dlJcEnXj62pPlwCmCV2pELJoEVx0Ebhc/jm+DtQV2DShKxUicnJg6VI7KJe/tK3Vlk2Zm9h9aLf/TqLOmiZ0pULEnj02mV98sf/O0T7Z/rbQZpfApAldqRBRqxbMmgVXXOG/c3RI7kB8+Xgmrpvov5Oos6YJXakQ4Xb7/xxRrij6nd+PaRumYYzx/wlVqWhCVypENG0KDz7o//N0q9uNjAMZbNi/wf8nU6WiCV2pELBtm32oqGZN/58rvz/60h1L/X8yVSqa0JUKAfkPFPnzhmi+RlUbESERrNi5wv8nU6WiCV2pEDBrFlSsCC1b+v9cFaMq0jqxNbM2z/L/yVSpaEJXKgSMGgWHDtmZispCvwb9mLd1HnsO7ymbEyqvaEJXKsjldzYZOrTsznlpo0txGzczN80su5OqM9KErlSQE7FJ/d13y+6crRNbExMZoxNHBxhN6EoFuUWLbA+XshTliqJjckemb5xetidWJdKErlSQ+8tf4JZbyv68lza6lLTdaWzJ2lL2J1fF0oSuVBA7dAjmz4cePcr+3Pn90ZdsX1L2J1fF0oSuVBCbP98+8t+1a9mfu3mN5kRIBMt36gxGgcKrhC4i/UVknYiki8hDJWzXVkTcInK170JUSp3O1Kl2dqKyeKCoqJjIGBokNGDlrpVlf3JVrDMmdBFxAW8AA4AmwHUi0uQ02z0HfO/rIJVSxZs2DTp3hkqVnDl/26S2zNkyhzyT50wA6iTe1NDbAenGmA3GmBxgHFDcAJ33ABOAXT6MTylVgqlT4a23nDv/gPMHsOvQLh3XJUB4k9CTgK2FPmd41p0gIknAIGBUSQcSkWEislhEFu/erTOeKHWuqlaFCy5w7vy9UnoB6ANGAcKbhC7FrCs6EPLLwIPGmBJHZDbGjDbGtDHGtKlWrZqXISqlivPSSzB6tLMxJMYm0rhqYx3XJUB4k9AzgNqFPicD24ts0wYYJyKbgKuBN0XkSl8EqJQ6lTHwwgswY4bTkdhZjBZuW6gTXgQAbxL6IqChiKSISBQwBPiq8AbGmBRjTD1jTD1gPHC3MWair4NVSllpabBjB/Tt63Qk0D6pPbsO7WJz1manQwl7Z0zoxphcYAS298oa4DNjTJqIDBeR4f4OUCl1qqlT7bJPH2fjgIKJoxdkLHA4EuXVYJvGmMnA5CLrir0BaowZeu5hKaVK8v33cOGFULv2mbf1t+bVmxNdLpqF2xZybbNrnQ4nrOmTokoFmfym6ksucTaOfJGuSFontubnjJ+dDiXsldFw+EopXxGxNfRAugfZKbkTI+ePJMedQ5QryulwwpbW0JUKMm5P52AprkOxQ1JrpeI2btJ2pTkdSljThK5UkOnQAe67z+koTtalThcApv421eFIwpsmdKWCyM6dsHgx1KjhdCQnS45LJqVyCkt26FC6TtKErlQQGTfOLvv1czaO4nRI7sCPm37EnVfiA+PKjzShKxVEJk2yy4sucjaO4lzR+Ar2HN7Dwm0LnQ4lbGlCVypIHDgAc+bA//1fYN0QzdenQR8E4YcNPzgdStjShK5UEHnuObjxRqejKF6VClVondiaaRumOR1K2NKErlSQiIuzE0K3auV0JKfXu35v5mfMJzsn2+lQwpImdKWCwL59MHYsHDzodCQl61O/D8fzjjNrkw6n6wRN6EoFgS+/hFtugfXrnY6kZJ3rdCa6XLS2oztEE7pSQeCOO+yydWtn4ziT6HLR9ErpxaR1k5wOJSxpQlcqwG31TADZs2dg9m4pql+DfmzK3MSmzE1OhxJ2NKErFeA+/NAu33zT2Ti81b1edwBtR3eAJnSlAtzatdC1KzRu7HQk3mlavSlVK1TVeUYdoMPnKhXg3n8fsrKcjsJ7ERLBxXUvZuammU6HEna0hq5UANu/3y7j452No7S61+vOxsyNbMna4nQoYUUTulIB6tdfITERJkxwOpLS61a3G6Dt6GVNE7pSAerVVyEvDzp3djqS0mteozkJ0Qna7FLGNKErFYCysuDdd+H666FmTaejKb0IiaBbvW7M3DzT6VDCiiZ0pQLQ3XfDoUMwYoTTkZy9bnW7sWH/BrZmbXU6lLChCV2pAJOXBx9/bN+3aeNsLOfiRH907b5YZjShKxVgIiJgyxZIC/L5llvUaKHt6GVM+6ErFUDcbnC5oHZtpyM5dxESQde6XbWGXoa0hq5UAHnmGejdG44edToS3+hetzvp+9LJOJDhdChhQRO6UgFi/3544QWoVAmio52Oxje61dP+6GVJE7pSAeLJJ+28oU884XQkvtOyRkviy8drO3oZ0YSuVABYvx5eew1uvx1atHA6Gt9xRbi0Hb0MaUJXKgA88wxUqAD//rfTkfhe97rd+XXfr2zO3Ox0KCFPE7pSAeC11+Dbb6FGDacj8b1BFw4C4OOVHzscSejThK6Ug/bvh//+FyIj7Zjnoah+Qn1a1mjJ9I3TnQ4l5GlCV8ohxsBtt8GwYfbp0FDWMbkjC7ct5Lj7uNOhhDRN6Eo55L33YOJEeOop234eynrX783BnIMs3LbQ6VBCmlcJXUT6i8g6EUkXkYeKKb9BRFZ4XvNEpKXvQ1UqdPz8s62d9+wJDz7odDT+1zOlJxESwdTfpjodSkg7Y0IXERfwBjAAaAJcJyJNimy2EehmjGkBPAmM9nWgSoWKo0ehUyf7/oMP7KP+oS6hQgJta7Vlcvpkp0MJad7U0NsB6caYDcaYHGAccEXhDYwx84wxnsmymA8k+zZMpUJHVha0bWuTea1aTkdTdvqf358l25eQdTSIJkgNMt4k9CSg8IDGGZ51p3M7MKW4AhEZJiKLRWTx7t27vY9SqRCRl2e7Ji5YADfe6HQ0ZatHvR4YDNM2THM6lJDlTUKXYtaZYjcU6YFN6MW2ChpjRhtj2hhj2lSrVs37KJUKAU89Bamp9vF+Ke7/qhDXqXYnalaqyXvL33M6lJDlTULPAAoP5pkMbC+6kYi0AP4HXGGM2eub8JQKDZ98Av/4B1SsCOXLOx2NMyJdkdzU4ia+S/+OvYc1RfiDNwl9EdBQRFJEJAoYAnxVeAMRqQN8AdxkjFnv+zCVCl7/+5+dG7RDB/jhh/BN6ADXNr2W3LxcJq2b5HQoIemMCd0YkwuMAL4H1gCfGWPSRGS4iAz3bPYYUBV4U0SWichiv0WsVBB57z24806oWhW++y70+5ufSevE1tSOq82naZ86HUpI8qofujFmsjGmkTGmgTHmKc+6UcaYUZ73dxhjEowxrTyvIJ4JUSnfufhim9C3boX4eKejcZ6IcFOLm5i2YRrbDmxzOpyQo0+KKuVjmzfDo4/aR/vr14fRo7VmXthtF92GMYbXFr7mdCghRxO6Uj703XfQujW8+iqkpzsdTWBqUKUBgy4cxDu/vIM7z+10OCFFE7pSPpCZCXfcAQMGQFISLFkCDRs6HVXgurbptew5vIc5W+Y4HUpI0YSulA8MHAjvvmvHZVmwAM4/3+mIAtslDS8hJjKGcavGOR1KSNGErtRZmjMHDh2y7x9/HBYtgmef1fZyb1SMqsjljS9n/Jrx5OblOh1OyNCErlQp/fILXHqpnZDi7bftuj59bNu58t6QpkPYc3gPMzbOcDqUkKEJXSkvGAOzZ8NVV9nH9+fOtbXx4cPPvK8qXv/z+xNfPl6bXXxIE7pSJcjOLnj/6KMwaxY88ghs2mTby2NiHAst6JUvV54rL7iSL9Z8wbHcY06HExI0oStVRE6OnUnoD3+AatUgI8MOpvXJJ/YBoSef1IeEfGVIsyFkHctiSnqxA7SqUtKErpTHzp3wpz9BYiIMGmRveg4fXjAyYmKi1sh9rVdKL6rFVOPdZe86HUpI0ISuwpLbDUuXwgsvwFeeoeaMsV0P+/aFb7+Fbdtg5Ejbr1z5R6QrkuFthvPVuq9YuXOl0+EEPU3oKqy8/XZBU0pqKvz1r/bpToCaNWHXLtu0MnAglCvnbKzh4r4O91EpqhJPzX7K6VCCniZ0FXLy8mD9ehg3zt64vO++grJ337VPcQ4aBB9+aGvhb75ZUF6pUpmHG/aqVKjCiLYj+CztM9bsXuN0OEFN6yAqKBlj27w3b4YtW2DwYLv+4Yfh9dcLeqdERkL79nZ7Efj+e4iLC88ZgwLZ/R3v59WFr/L0nKf5YNAHTocTtLSGrgJOXh7s3g2rVsGMGfDxx3DkiC373//gggvszcnERDtpxDXX2ImXARo3hqFDYcwY+wBQdrbtP56fwOPjNZkHomoVq3FXm7v4eOXHrN2z1ulwgpYYU+z0oH7Xpk0bs3ixzoMRqoyBw4fh4EH7Sky0zRlbtsC0aTYBZ2XB3r2wZ4/tCnj++bZJ5I47bFIvbNUqaNoUxo+HTz+FevXsq25dSEmxiVzbvIPbzuydNHytIQMbDmTc1fqw0emIyJLTzTmh/wsEMWNs4svNhYgI27xgjK3dut12fW6ufV+5Mpx3nu1jvWSJXRZ+NWlik2JWlr0pWLjs2DG45BJo1w42boR//tPWmAu//vUv2ztk9mz7WHx29slJ+Ztv7DGWLYPbby9Yn5Bg49q3z35u2RL+/neoXh1q1ChYNmhgy6++2r5U6KlRqQZ/TP0jI+ePZOP+jaQkpDgdUtAJyoT+zju2LdSYgqQWGWlrbgAvvWT/VM/LK9gmPr6g/LHHbB/jwuWJiQXl994LCxeeXN6woU10ADfeCCtWnFx+0UXw0Ue2fMAAe1Muv8wYO3PN2LG2vG1b+7BK4fLLLrPfC2zNc//+k8tvugneesuWx8TYJFs4Yf75z/Dyyza51qhx6jX7xz9sLTgzEzp1OrX82WftDcQ9e+Cuu04tr1nTJvQjR2zSrlABoqNtLBUrgstlt0tKgltvhdhYWyOPjbWvli1tec+e9pdCfLxty87fL1/r1jomSjj7c4c/88aiN3jghweYcM0Ep8MJOkGZ0Ldvh5UrbVtoRIRdRkcXlO/fD7//btfnvwrLr7nml0VEnPznekyMrdEW3v+88wrKk5Jsc0Lh/fNrkAAtWtg5JAuXN29eUN6zp42xcHnhJDZkCBw9enJ527YF5X/5i13vctm4Xa6C8vLl7U3B/PXlytlXs2a2vHJl200vKurkV61atrxuXdixo2B9ZKRd5l/DJk1sQj6d+vXtL5bTqVRJe5Ko00uOS+b+jvfz1OynWLJ9Cam1Up0OKahoG7pSKqBkHc2iwasNaF6jOdNumoYrwnXmncJISW3o2stFKRVQ4qPjebLHk8zcNJPHfnzM6XCCiiZ0pVTAGd5mONc1u46n5zzN5F8nOx1O0NCErpQKOCLCmCvG0Kx6M+7+9m4dXtdLmtCVUgEpulw0L/R5gc1Zmxn2zTCcut8XTDShK6UCVr/z+/FI10cYu3wsT8x6wulwAl5QdltUSoWPJ3s8yfaD2/nXrH9RK7YWd6be6XRIAUsTulIqoIkIb1/6NlsPbGXYN8NIjktmQMMBTocVkLTJRSkV8CJdkUwaMokWNVow+PPBzNw00+mQApImdKVUUIiJjOG7G76jVmwteo/tzcifR5Jn8s68YxjRhK6UChqJsYksunMRAxoO4P6p93P3t3drUi9EE7pSKqjER8fz1ZCvuKfdPby95G1GTB7B/iP7nQ4rIGhCV0oFHRHhlf6vcHPLm3lr8Vt0fKcjP278Mez7qmtCV0oFJRHh/SvfZ+qNU8nOyabn2J5c9sllzNw0M2ybYTShK6WCWp8GfVg7Yi2PXvwoMzbOoMf7PWj2ZjPeWvQW2w9udzq8MqXD5yqlQsbh44eZsHoCL/78Ist3LkcQutTpwg3Nb+DSRpeSFJfkdIjnrKThc71K6CLSH3gFcAH/M8Y8W6RcPOUDgcPAUGPM0pKOqQldKeUvxhiW71zO+NXj+XLtl6zevRqA5tWb06VOFxIrJZIYm8j5Vc6nUdVGJFZKRIJk9vBzSugi4gLWA32ADGARcJ0xZnWhbQYC92ATenvgFWNM+5KOqwldKVUWjDEs3LaQeVvnnUjue4/sPWmbipEVaVi1IbXjahNXPo648nHERsXaZfnYYtdFRkTiinDhEhcREnHivSvC87mY9y5xnfMvjnOdJLodkG6M2eA52DjgCmB1oW2uAMYa+9thvohUFpFEY8yOc4pcKaXOkYjQPrk97ZPb85eOfwHgWO4xth/cTvq+dNbvXW9f+9azJWsLB3MOcvDYQQ4cO8Axt3+G7X2o80M80/sZnx/Xm4SeBGwt9DkDWws/0zZJwEkJXUSGAcM8H7NFZF2poj075wF7yuA8gUyvgV4D0GsAAXINnvX8d5bqnq7Am4Re3N8HRdtpvNkGY8xoYLQX5/QZEVl8uj9PwoVeA70GoNcAQv8aeNNtMQOoXehzMlC0L5A32yillPIjbxL6IqChiKSISBQwBPiqyDZfATeL1QHI0vZzpZQqW2dscjHG5IrICOB7bLfFMcaYNBEZ7ikfBUzG9nBJx3ZbvNV/IZdamTbxBCi9BnoNQK8BhPg1cOzBIqWUUr6lj/4rpVSI0ISulFIhIigTuojcIyLrRCRNRP5TaP3DIpLuKetXaH2qiKz0lL3qGaoAESkvIp961i8QkXqF9rlFRH71vG4p0y/oJRF5QESMiJxXaF1YXAMReV5E1orIChH5UkQqFyoLi2tQGiLS33M90kXkIafjORciUltEfhSRNZ4c8GfP+ioi8oPn3+oHEUkotI/PfiYCmjEmqF5AD2AaUN7zubpn2QRYDpQHUoDfAJenbCHQEdtffgowwLP+bmCU5/0Q4FPP+yrABs8ywfM+wenvXuQ61MbeqN4MnBdu1wDoC5TzvH8OeC7crkEprpXLcx3qA1Ge69PE6bjO4fskAq0972OxQ5M0Af4DPORZ/5A/fiYC/RWMNfS7gGeNMccAjDG7POuvAMYZY44ZYzZie9y0E5FEIM4Y87Ox/zpjgSsL7fO+5/14oJfnN3Q/4AdjzD5jzH7gB6B/GXy30hgJ/I2TH+AKm2tgjJlqjMn1fJyPffYBwugalMKJ4TuMMTlA/vAdQckYs8N4Bv8zxhwE1mCfTC/87/g+J//7+upnIqAFY0JvBHT1/Bk0S0TaetafbviBJM/7outP2seTHLKAqiUcKyCIyOXANmPM8iJFYXMNirgNW7uC8L0GJQmV73EKT1PIRcACoIbxPP/iWVb3bObLn4mA5s2j/2VORKYBNYspegQbcwLQAWgLfCYi9Tn98AMlDUtwNvuUiTNcg79jmxxO2a2YdSF5DYwxkzzbPALkAh/l71bM9kF7DXwkVL7HSUSkEjABuM8Yc6CECrQvfyYCWkAmdGNM79OVichdwBeeP5EWikgedsCd0w0/kEHBn+OF11NonwwRKQfEA/s867sX2Wfm2X+j0jvdNRCR5th2wOWeH+BkYKmItCNMrkE+z03KS4Fenp8HCLFr4CMhNzSHiERik/lHxpgvPKt3imeUV09zSn5zrC9/JgKb0434pX0Bw4EnPO8bYf8sEqApJ9/42EDBjY9F2Bp9/o2PgZ71f+LkGx+fed5XATZi/xJI8Lyv4vR3P8312ETBTdGwuQbYtuzVQLUi68PmGpTiWpXzXIcUCm6KNnU6rnP4PoJt7365yPrnOfmm6H98/TMR6C/HAziLf8wo4ENgFbAU6Fmo7BHsHex1eO5We9a38Wz/G/A6BU/IRgOfY2+SLATqF9rnNs/6dOBWp793CdfjREIPp2vgiWkrsMzzGhVu16CU12sgtjfIb9gmK8djOofv0gXb/LGi0L//QGwb93TgV8+ySqF9fPYzEcgvffRfKaVCRDD2clFKKVUMTehKKRUiNKErpVSI0ISulFIhQhO6UkqFCE3oKuiISFURWeZ5/S4i2zzvM0VktR/O9y8ReaCU+2SfZv17InK1byJT6mSa0FXQMcbsNca0Msa0AkYBIz3vWwF5Z9rf8+SfUiFHE7oKNS4R+a9nnOypIlIBQERmisjTIjIL+LNnHOxZIrJERL73PCqOiNwrIqs946yPK3TcJp5jbBCRe/NXisj9IrLK87qvaDBive455rcUDBillM9pTUWFmobAdcaYO0XkM+AP2CeLASobY7p5xgGZBVxhjNktItcCT2GfCn0ISDHGHJNCk2YAF2DH4o8F1onIW0AL7ITo7bGPji8QkVnGmF8K7TcIaAw0B2pghysY448vrpQmdBVqNhpjlnneLwHqFSr71LNsDDQDfvAMcOYCdnjKVgAfichEYGKhfb81dgz+YyKyC5ucuwBfGmMOAYjIF0BXoHBCvxj4xBjjBraLyIxz/4pKFU8Tugo1xwq9dwMVCn0+5FkKkGaM6VjM/pdgk/DlwKMi0vQ0xy1H8UOsFkfH11BlQtvQVThaB1QTkY5gh2IVkaYiEgHUNsb8iJ0NqjJQqYTj/ARcKSIxIlIR27wyu5hthoiIy9NO38PH30WpE7SGrsKOMSbH03XwVRGJx/5/8DJ2NMIPPesE23sm83QTJxhjlorIe9jR+AD+V6T9HOBLoCew0nP8WT7+OkqdoKMtKqVUiNAmF6WUChGa0JVSKkRoQldKqRChCV0ppUKEJnSllAoRmtCVUipEaEJXSqkQ8f8VrM+hYx9xfgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_precision_recall_vs_threshold(precisions, recalls, thresholds):\n",
    "    plt.plot(thresholds, precisions[:-1],'b--', label = 'Precision')\n",
    "    plt.plot(thresholds, recalls[:-1],\"g-\", label = \"Recall\")\n",
    "    plt.xlabel(\"Threshold\")\n",
    "    plt.legend(loc = \"best\")\n",
    "    plt.ylim([0,1])\n",
    "plot_precision_recall_vs_threshold(precisions,recalls,thresholds)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，在召回率在 80% 左右的时候，准确率急剧下降。你可能会想选择在急剧下降之前选择出一个准确率/召回率折衷点。比如说，在召回率 60% 左右的点。当然，这取决于你的项目需求。\n",
    "\n",
    "我们假设你决定达到 90% 的准确率。你查阅第一幅图（放大一些），在 70000 附近找到一个阈值。为了作出预测（目前为止只在训练集上预测），你可以运行以下代码，而不是运行分类器的predict()方法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAArZklEQVR4nO3dd3xV9f3H8dcni7DCMGFvZSOgREQBBS1WUIsiWnddRap2/NTW1dafWvfoFBFHrT9bJ2gdiFqr4kIJssGBjBDZK2EkQJLv7497E5KQcRPuueeO9/PxyCO5557c+zkG88k57/P9fs05h4iIJK4kvwsQERF/qRGIiCQ4NQIRkQSnRiAikuDUCEREEpwagYhIgvOsEZjZU2a2ycyW1PC8mdlfzGyFmS0ys6O9qkVERGrm5RnB08CptTw/FugZ/JgEPOphLSIiUgPPGoFzbjawrZZdxgPPuIA5QEsza+9VPSIiUr0UH9+7I7C2wuO84Lb1VXc0s0kEzhpo2rTpkD59+tT7zfIL95O7bU+lba2apNGpVeN6v5aISMSUlkD+WijcDk2zoEWnBr3MvHnztjjnsqp7zs9GYNVsq3a+C+fcNGAaQHZ2tsvJyan3mxXuK2FH4b7yx2Mens1Jfdrwu9P7sb+klA4t1RBEJMqs+QxmTIKCUhh1L4y8DpKSG/RSZrampuf8bAR5QOcKjzsB67x6s8ZpyTROO/DLvnl6Cq8tXMdrC9fRI7Mp/71hVNjf0zlHQVEx63YUsn3PPgZ3bsn2PfvZVFDEvuJSBndpSaOUhv1QRSSOlRTDh/fBRw9Cyy5wxTvQKduzt/OzEbwGXGtmzwPHAvnOuYMuC3llfX5R+dcrt+w+6Pl1OwpJMqNdi3RWbNrJp99tZe7q7cz+ZjMXDevCdWN6k5wUOKkpKNrPtxt38c3GnXyzcSert+xm3Y4ivt9RyK69xXXWMrp3FsOPyGRjQRG92jZnb3Epa7fv4ajOLdmQX0SbjHSSDDbv2kfTtGR+NKgDKcm681ckLm1bBTN+CnlzYdAFMO5+aNTc07c0r2YfNbPngFFAJrARuA1IBXDOTTUzA/5G4M6iPcBlzrk6r/k09NJQVbOWrKd5eioPvfM1X+buYPKJh/POsg1cN6YXr87/nv8s3wTAEW2asWLTLgDaZaSzoaCo0ut0bt2YtdsKyx83Tk2me2ZTOrVqTIeWjenYMvB59jeb6dCyMW0yGtGycSrvLNvIK/O/P6RjuGhYF0b2zMI5x0l92pKWUrk5lJQ6kpOMvcUlJJupeYhEM+dg0Qvw5g1gSXDGH2HA2WF7eTOb55yr9rTCs0bglXA1gjL3z/qKKR98V2lbZrM0tuwK5AnDerTmtCPbM6p3Gzq1asxdby4nZ812FqzdQZO0ZE7slcWAji3o3bY5vds1p2PLxiQlVRd/VM85R+62PaSnJrOxoAjDaN0sjbXb9tCsUQrbdu/DDFo3TaNofwnXvbiQvO2FlJQe/HNrkpZM/w4ZbN65l9xte6hmFy4f3p2vNhTQq21zurRuQrNGKRQU7eeoLq0Y0rVV/f7jiUh4FO6AN6+HJS9Dl+NhwmOBS0JhpEZQi1fm5/E/LyzkqhN7MH3e95w5uAPXndKLJDN27S0ms1mjar+v7L9b4MQm8nbs2cecldtonJbMdS8sYOvufXTPbEq7jHSymjcio3EK32zYRb8OGWzaWcS7yzayv6Tun3XzRins3FtMWkoS+4pL6XZYE7bu2seYfm05slMLBnduycBOLcsvi4nIISoPhL+HUTcfUiBcGzWCWjjn2Lm3mIz01LC9ZrTaX1JKanISW3btJW97Ic45UpKSeGfZBp77IpejurQiv3A/Gekp/Gf5JrpnNmVVNfkJQPsW6azPL2L84A4sX19AWkoSHVo0puthTRjUuSUDOrRg6+69rNtRRNNGyYzsmUWqLk2JHFA1EJ7wBHQ+xrO3UyOQQ1JcUsqy9QWs2rKbXz6/4JBfr1fbZvTIbMYx3Vvzk+O6KruQxFM1EB57H6RnePqWagQSEd/vKGTZugK+2biTdhnpHNYskGu8lJNH3vZCvt64s9rv69c+g0GdWzCqdxt6ZDbliDbNfLvkJuIpjwPh2qgRSFTZs6+YvO2FnP7Xj0lLTqr2FtvMZmlkNU8nLdm4fER3TurThuYJcPlO4lhRPrxxXTAQPg4mTAt7IFwbNQKJas45vszdzvr8Iqa8/x35hfvZunsvRftLK+2XkZ7CQ+cOplWTVPp1yKBJmp/DYETqIUKBcG3UCCRmfb1hJ28tWc+f/vPtQc+d3KcNN43tQ8+23g62EWmwCAfCtVEjkJjnnGNhXj4b8gv594J1vLVkQ6Xnjz/8MAZ2asnhWU05umsremQ2Vc4g/qoUCJ8PY+/3PBCujRqBxB3nHDMXb+CmGYvYWVT9NB5N0pLpkdWUod0O4/IR3chs1oj0VM3tJB6rGgif/jAcOdHvqtQIJP4V7S9h9dbdfL1hJy/mrGXl5t2kpSSxZuueavfv3bY5J/dtwzWjj6BpI2UNEiY+B8K1USOQhFW4r4R/fZHLmq272VVUzIxq5ndqkpbM+MEduHP8AI1pkIbLnQPTf+prIFwbNQKRKgr3lXD9Swt4Z+lGiitMyjSwUwvat0jnnCGdadU0TfMvSd1KimH2/TD7Ad8D4dqoEYjUYl9xKb1++1aNz182vBunD2zPkK6tI1iVxIQoC4Rro0YgEqL9JaW8vnAdzRqlMOn/5h30fJ92zemR1ZTde0u4eFhXftCvrQ9VSlRY+EJgxtAoCoRro0Yg0kD7S0qZtWQDP39ufo379GnXnBlXH68BbomiKD/QABa/FHWBcG3UCETC7NMVW7jgic8rbRvUuSWXDOvKmUd11DTd8SrKA+HaqBGIeMQ5x8SpnzFvzfZK2/u1z2BI11ZcMaI73TKb+lSdhE2MBMK1USMQiYAVm3Zx1pRPqh3gdu+EIxk7oD0tmmjivJizbVVgnqC8L6I+EK6NGoFIhO3Ys48Xc9Zy98yvDnruVz/oydlHd6Jz6yY+VCb1EmOBcG3UCER8tG5HIU9+vIonP15VafukE3pwy7i+PlUltYrRQLg2agQiUWLzzr1c868v+WLVtvJtV4zozpUju9O+RWMfK5NyMRwI10aNQCTKLF9fwNg/f1RpW6OUJEb2zGTEEZn85Phumj010ioGwi06w9lPxlwgXBs1ApEotXbbHh6b/R3Pzsk96LkeWU35yXHduGhYV92O6rXtqwNnATEeCNdGjUAkRhTuK+GGlxfy5qL15dt6ZDZl6sVD6KUFeLwRR4FwbdQIRGKMc478wv0MvuPd8m2NUpK49PhuXD6iO20z0n2sLk7EYSBcGzUCkRg2b812Jj2Tw9bd+8q33T9xIGcd1ZFUTZvdMLlzApPF5X8Po26CEddBcnxPEaJGIBIHivaX8NA7X/P4R5VvQ1142ym0aKyBaiEpKQ6EwbPvj8tAuDa1NQL9OSESI9JTk7n1tH5M/9nx/KBvm/Ltg25/h6Xr8n2sLEZsXw1/Hwsf3gtHnguTP06YJlAXnRGIxCjnHN1vnllp270TjuS8ofF7nbvBEiQQro3OCETikJmx+t7TeOSCo2ndNA2Am2Ys5rh73mNvcYnP1UWJonyYfiW8MgnaDYCffZyQTaAuOiMQiRNf5m5nwpRPyx+fdmR7fn7yEfRpF1/3w4csAQPh2igsFkkQ+0tKOfqOd9m5t/IMqK9fO4IjO7XwqaoIOygQfgI6D/W7Kt+pEYgkGOcc89ZsZ+LUzyptj/uJ7iqOEB54Hox7IO5GCDeUGoFIAvtkxRYurLCaWvP0FD6+8aT4u+W0PBA2OP2PygKqUFgsksCGH5HJ6ntP4z/XnQjAzqJiBt3+Dhvyi3yuLEyqBsKTFQjXl6eNwMxONbOvzWyFmd1UzfMtzOx1M1toZkvN7DIv6xFJZEe0acbKu8cxuncWAMPueY8nPlpJrF0VqCR3DkwdAUtmwOhb4SdvQKuuflcVczxrBGaWDDwCjAX6AeebWb8qu10DLHPODQJGAQ+ZWZpXNYkkuqQk46lLj+Hy4d0B+MOby+l+80xufHkRhfti6JbTkmJ4/57AADEMLp8FJ/4moe8KOhRenhEMBVY451Y65/YBzwPjq+zjgOYWmHi9GbANOHjBVxEJGzPj92f046PfjObY7q0BeCFnLX1/P4sbX15E0f4obwjVjhDWXUGHwstG0BFYW+FxXnBbRX8D+gLrgMXAL51zpVVfyMwmmVmOmeVs3rzZq3pFEkrn1k144arj+PauseXbXshZS5/fzeLpT1bV8p0+WvgCPDoCNn8VmCdowmO6KygMvGwE1a2kUfVi5A+BBUAHYDDwNzM76KfqnJvmnMt2zmVnZWWFu06RhJaanMTqe09j2R0/5LgehwHwv68vO2iNZV9VDITb9lcgHGZeNoI8oHOFx50I/OVf0WXADBewAlgF9PGwJhGpQZO0FJ6bNIwXrzoOgDvfWManK7b4XBUHB8KXvqlAOMy8bARzgZ5m1j0YAJ8HvFZln1zgZAAzawv0BlZ6WJOI1GFo99Y8fO4gAC544nO+31HoTyEKhCPGs0bgnCsGrgXeBpYDLzrnlprZZDObHNztTuB4M1sMvAfc6JyLgj9BRBLbhKM7cenx3QAYfu9/+cenqyNbwPbV8PQ4BcIRopHFIlKjv773LQ+9+w0AKUnG3Ft/QKumHt/hvejFwAhh0AjhMNLIYhFpkJ+f3JOnLwss3lJc6jjqznf57auLvbnFtCwQnvFTaNNPgXAEqRGISK1G9W7DqnvGMSo4IvnZObn0+d2s8L5J7ucHAuFRtygQjjA1AhGpk5nx9GVDWX3vaeXbut30Ju9/tenQXrg8ED6V8kB41I0KhCNMjUBE6mXR/55S/vVlT8/lqYaON1AgHDXUCESkXjLSU1l972ncGlzX4I43lnHxk5+zr/igSQFqtuhFmDoSNi3XCOEooEYgIg3y0xN68LvTA/NIfvTtFnr99i0+/KaOKWCK8gMLxygQjipqBCLSYFeM6M6qe8Zx8bBAsPuTp77gmn9+Wf3O5YHwdAXCUUaNQEQOiZlx55kDePaKYwF4c/F6rv7nvAM7KBCOemoEIhIWI3pm8vq1IwCYuXgDZz7yiQLhGKFGICJhc2SnFuV3FXX9/g32PzJcgXAM0LmZiIRVBnuY2/cFslb9m7n7erF46INcfuSJfpcltVAjEJHwyf0cZlxJVv73fNFtMud/NZySj3fx+fYcHru42mluJAro0pCIHLpqAuGhl97HK9eeAMDbSzfyr89z/a1RaqRGICKHZvuaCoHwOZUC4YGdWvLS5MBCN7e8spizpnziZ6VSAzUCEWm4RS8GxgZsWg4TnoAJ0w4KhI/p1ppZvxoJwPzcHeSs3uZHpVILNQIRqb/qRggPPKfG3fu0y+DVa4YDMHHqZ0yb/V2kKpUQqBGISP2UjxB+GUbdHPII4cGdWzLt4iEA3D3zK37/7yXsL6nH/ETiGTUCEQlNSTF8cO+BNYQvmwWjbqrXCOFT+rfjwXMC6yE/89kaet76FnuLPVjkRupFjUBE6lYWCH9wT2CSuMkfQ5djG/RSE4d0YuFtB6ay7v3bMC9yI/WmRiAitVv0Up2BcH21aJzKd3ePK3/c4+Y3ibX10+OJGoGIVK8oH2ZMghlXQpu+MPmjWgPh+kpOMj67+SQASh10v3km+YX7w/b6Ejo1AhE5WFkgvPilYCA8E1p1C/vbtG/RmJV3j6Nf+8AZxsRHPw37e0jd1AhE5IBKgTANCoTrKynJmPnLwDiDbzftUjPwgRqBiARsXwNPnxaWQLgh/n7ZMQDkrNnOL56bH7H3FTUCEYEKgfCyCoFwi4iWMLp3Gz6/5WQAXlu4jtl1LXspYaNGIJLIigo8DYTrq21GOk9dGpil9JKnvmDrrr2+1ZJI1AhEElWEAuH6OqlPW9pmNAJgyB/+Q0mpbiv1mhqBSKKpFAi7iATC9fX5LT8gJckAOOqOd3yuJv6pEYgkkoqB8ICzIx4I18f8348BoKComFlLNvhcTXxTIxBJFJUC4cfh7McjHgjXR/P0VP583mAAJj87j6c/WeVvQXEses4FRcQbRQUw8wZY9AJ0PjZwR1AUZAGhGD+4I41Skpn87Dz+9/VltMlIZ9yR7f0uK+7ojEAknq39IioD4fo4dUA7/vTjwQBc/c8vmbVkvb8FxSE1ApF4VFIMH9wHT51KtAbC9XHmUR15/JLAbaWTn/2SHXv2+VxRfFEjEIk35YHw3VEfCNfHmH5tOWNQBwAG3/Guz9XEF08bgZmdamZfm9kKM7uphn1GmdkCM1tqZh96WY9I3CsLhDcujYlAuL7+EgyPAeas3OpfIXHGs0ZgZsnAI8BYoB9wvpn1q7JPS2AK8CPnXH/AvyGNIrGs6gjhn30MA8/1u6qwMzOevSJwdnPetDl8u3GnzxXFBy/PCIYCK5xzK51z+4DngfFV9rkAmOGcywVwzm3ysB6R+BQHgXB9jOiZyflDOwMw5o+zfa4mPnjZCDoCays8zgtuq6gX0MrMPjCzeWZ2SXUvZGaTzCzHzHI2b9ZEVCJA3AXC9XHPhIF0z2wKwF1vLvO5mtjnZSOwarZVnTQkBRgCnAb8EPidmfU66Jucm+acy3bOZWdlZYW/UpFYE6eBcH08etHRADz+0Soe/eA7n6uJbV42gjygc4XHnYB11ewzyzm32zm3BZgNDPKwJpHYF+eBcKj6tMsoX8PgvllfMXf1Np8ril0hNQIzG25m75rZN2a20sxWmdnKOr5tLtDTzLqbWRpwHvBalX3+DYw0sxQzawIcCyyv70GIJIQECYTrY3TvNjz5k8D4gnOmfsY3Co8bJNQzgieBh4ERwDFAdvBzjZxzxcC1wNsEfrm/6JxbamaTzWxycJ/lwCxgEfAF8IRzbklDDkQkriVYIFwfJ/dty63j+gJw/6yvfa4mNplzdc/1bWafO+ei4gJkdna2y8nJ8bsMkcgoKYaPHoIP74MWHQOrhyVYFhAK5xzdb54JwHd3jyM5qbqIMrGZ2TznXHZ1z4V6RvC+mT1gZseZ2dFlH2GsUUSqUiAcMjNjTL+2AEyY8onP1cSeUO8zK/vXV7GbOOCk8JYjIkAgEH7zOnAuEAgneBYQigfPGcSg299hYV4+q7fsplvw9lKpW0hnBM650dV8qAmIhFvFQDirjwLhemjROJUnghPTjXrwA3+LiTEhnRGYWQvgNuCE4KYPgTucc/leFSaScNZ+AdOvhPy1gUB45A0JMTgsnH4QvDwE6KygHkLNCJ4CdgLnBj8KgL97VZRIQkngEcJe+NsFRwEw/hFlBaEKtREc7py7LThv0Ern3O1ADy8LE0kICoTD7vSBHejSugn5hft5Z6nWOg5FqI2g0MxGlD0ws+FAoTcliSSIxS9rhLBHyqafmPR/81i1ZbfP1US/UBvBz4BHzGy1ma0B/gZM9q4skThWFghPv0KBsEf6d2jBVScGLlqMfvADtu7a63NF0S3Uu4YWOOcGAQOBI51zRznnFnpbmkgcqjhC+MSb4LK3NELYIzeP7cu4I9sBcNw9//W5muhWayMws4uCn68zs+uAK4ErKzwWkVAcFAi/BaNvViDssSkXDmFw55bsKynlX5/n+l1O1KrrjKDs3qvmNXyISF2qDYSH+V1VwvjLeYG7iG55ZTGhTKmTiGr9c8Q591jw8+2RKUckzix+Gd74n8AI4bOmwaAf+11RwulyWBMmDunEy/PyuPON5fz+jH51f1OCCXUa6vvNLMPMUs3sPTPbUnbZSESqUVQAM66qHAirCfjm5ycdAcC/vljjcyXRKdS7hk5xzhUApxNYTKYX8GvPqhKJZeWB8IsKhKNE18OaMrJnJkX7S3nwbU1VXVWojSA1+Hkc8JxzTksBiVRVWgIf3h8IhJ0C4Whzwym9Afjb+yso2l/iczXRJdRG8LqZfUVg9tH3zCwLKPKuLJEYUxYIv38XDJgQuBSkQDiqDOrckjMHdwCgz+9mKTiuINRxBDcBxwHZzrn9wG5gvJeFicSMshHCG5YEAuGzn9AI4Sj1xx8PLv964tTP/CskytR6zmpmJznn/mtmEypsq7jLDK8KE4l6RQUw89ew6HnoNBQmTIPW3f2uSmphZnx39zgOv2Um89ZsZ29xCY1Skv0uy3d1nRGcGPx8RjUfp3tYl0h0WzsXHhtZORBWE4gJyUnGSX3aAPDn/3zrczXRIaQ1i6OJ1iwWX5WWBNYQ/uBeyOgYmChOWUDM2VRQxNC73wPg27vGkpocalwauw55zWIzu9vMWlZ43MrM/hCm+kRiw45cBcJxok1GOhcP6wpAz1vforQ0tv4gDrdQ2+BY59yOsgfOue0EbiUVSQyLX4ZHFQjHkzvPHFD+9fD7EntSulAbQbKZNSp7YGaNgUa17C8SHyqNEO4Nkz/SCOE4suqewN+z6/OL2FiQuHfEh9oIniUwfuAKM7sceBf4h3dliUSBSoHwjQqE45CZ8dA5gwC44PE5Plfjn1DHEdwP/AHoC/QH7gxuE4k/5SOEfwilpcERwrdohHCcOntIJwC+27yb/ML9Plfjj/pE5cuBWc6564GPzEzTUEv8USCckMb0axv4/PCHPlfij1DvGvop8DLwWHBTR+BVj2oS8YcC4YQ17eIhAGzauZefPzff52oiL9QzgmuA4UABgHPuW6CNV0WJRFRRAbwyWYFwAjMzXr1mOACvL1xH/p7EukQUaiPY65zbV/bAzFKAxL7xVuJDWSC86AUFwglucOeW3D9xIACD7niH4pJSnyuKnFAbwYdmdgvQ2MzGAC8Br3tXlojHqgbCl85UICycm92ZEUdkAnDErW/5XE3khNoIbgQ2A4uBq4CZwG+9KkrEUxUD4f5nBS4FdT3O76okSjxz+dDyr7/duNPHSiKnzkZgZknAYufc4865c5xzE4Nf69KQxJ6qgfDEJ6FxS7+rkiiSlGRMufBoAP74n298riYy6jwPds6VmtlCM+vinMuNRFEiYbd3Z2DK6IXPQadjYMLjygKkRj/s3w6AmYs3+FxJZIR6aag9sDS4cP1rZR9eFiYSNmvnBhaOKQ+EZ6kJSK2Sk4yxAwLN4C/vxf9U1aEmY7d7WoWIF6pOGX3pTGUBErL7Jw7krSUbePjdb/jFyT39LsdTtZ4RmFm6mf0KOAfoA3zinPuw7KOuFzezU83sazNbYWY31bLfMWZWYmYT63sAItVSICyHqHl6avkaxzMXr/e5Gm/VdWnoHwQWrF8MjAUeCvWFzSwZeCT4ff2A882sXw373Qe8Hepri9SquhHCCoSlAa4b0xuAq//5ZVwvdl9XI+jnnLvIOfcYMBEYWY/XHgqscM6tDA5Ge57qF7z/OTAd2FSP1xY52N6dFUYI9zowQrjyOtsiIetyWJPyr7vfPNPHSrxVVyMoH2ftnCuu52t3BNZWeJwX3FbOzDoCZwFTa3shM5tkZjlmlrN58+Z6liEJQYGweGT5HaeWf33C/e/7WIl36moEg8ysIPixExhY9rWZFdTxvdX9GVb13OpPwI3OuZLaXsg5N805l+2cy87KyqrjbSWhaISweKxxWjJf/m4MALnb9sTlspa1/t/inEs+hNfOAzpXeNwJWFdln2zgeQucumcC48ys2Dn36iG8rySKHbkwYxLkfgYDJsJpDykLEE+0bprGqf3bMWvpBh798DuuGX2E3yWFVX3WI6ivuUBPM+tuZmnAeUClsQfOue7OuW7OuW4Eprm+Wk1AQlIpEH5MgbB47oFzAhPSPfD21+wtrvUiRszxrBEEM4VrCdwNtBx40Tm31Mwmm9lkr95X4ly1gfB5CoTFc83TU8sXsLlp+mKfqwkvi7VborKzs11OTo7fZYgf8nICDWBHLpzwazjhN8oCJKKcc/S/7W327Cvh27vGkprs5UWV8DKzec657Oqei52jkMRVWgIfPgBPnhL4WoGw+MTMOO+YLgDc+PIin6sJHzUCiW7lI4T/EBwh/LFGCIuvfnNqYJDZjPnfs6mgyOdqwkONQKKXAmGJQumpyTx0ziAArnwmPi5TqxFI9FEgLFHurKMCY2MX5eWzZ199x9pGHzUCiS55OQdGCJ/wG60hLFEpKcm4ZvThANw9c7nP1Rw6NQKJDtUFwifdCsmpflcmUq0bTglkBc/OyY350cZqBOK/Hbnw9OnBQPhMBcISE8ysfFzBvbO+8rmaQ6NGIP4qD4QXBwNhrSEsseOv5x8FwNzV23yu5NCoEYg/9u6EV36mQFhiWnpqMl1aN2F+7g4WrN3hdzkNpkYgkVceCD+vQFhi3vWn9ALgzEc+8bmShlMjkMg5KBB+U4GwxLzxgzvSs00zAB55f4XP1TSMGoFExo611QTCx/tdlUhY/POnxwKBmUljkRqBeG/JdHh0OGxYBGdOVSAscadN83R6tQ2cFbyxqOqyK9FPjUC8UxYIv3z5gUB48PkKhCUuPXzuYACu/dd8fwtpADUC8UZeDkwdWSUQ7uF3VSKeGdCxBacNbA/Aq/O/97ma+lEjkPAqLYHZZYFwsQJhSShXjwpMO/GrFxb4W0g9qRFI+JQFwv9VICyJqX+HFgzomAFATgwNMlMjkPBQICwCwO0/6g/EVlagRiCHRoGwSCVDurbGDDYUFPHe8o1+lxMSNQJpOAXCItWa+YuRAFzxj9hYuEaNQOpPgbBIrfq2z6DbYU0A+GTFFp+rqZsagdSPAmGRkDz848EAXPjE5/4WEgI1AgmdAmGRkB3dpRWtmgTOkpd8n+9zNbVTI5C6VQyEM3sqEBYJ0aMXDQHg9L9+7HMltVMjkNpVDYQvn6VAWCREw3ocRp92zQH44OtNPldTMzUCqZ4CYZGw+PN5gVXMLv37XJ8rqVmK3wVIFNqxFl65CtZ8AgPOhtMeVhYg0kC92zUnNdnYX+JYtWU33TOb+l3SQXRGIJUtmQFTh8P6hQqERcJkyoWBrGD0gx/4W0gN1AgkoDwQvgwOUyAsEk5j+rWlb/vAHETL1xf4XM3B1AgE8uZVCIR/rUBYxAO//mFgbeOxf/7I50oOpowgkZWWwMcPw/v3QEaHQCCswWEinhjdu03512u37aFz6yY+VlOZzggS1Y618I8zNEJYJELMjKcuzQZg5P3v+1xNZWoEiUiBsIgvTurTlk6tGgMwN4rWK1AjSCQKhEV8NzU42vicqZ/5XMkBnjYCMzvVzL42sxVmdlM1z19oZouCH5+a2SAv60loCoRFosKAjgdWMYuWOYg8awRmlgw8AowF+gHnm1m/KrutAk50zg0E7gSmeVVPwiofITymwgjh32qEsIiPrj+lNwBX//NLnysJ8PKuoaHACufcSgAzex4YDywr28E592mF/ecAnTysJ/FUHCHcfwKc/kdlASJRoOwOotxte3DOYT5fnvXy0lBHYG2Fx3nBbTW5AniruifMbJKZ5ZhZzubNm8NYYhyrGghPfEpNQCSKnD+0CwB//2S1v4XgbSOorsW5anc0G02gEdxY3fPOuWnOuWznXHZWVlYYS4xDe3fCq1crEBaJcjeeGrg89Njs73yuxNtGkAd0rvC4E7Cu6k5mNhB4AhjvnNvqYT3xrywQXvicAmGRKNeySRqXHNeVjQV7fV/k3stGMBfoaWbdzSwNOA94reIOZtYFmAFc7Jz7xsNa4psCYZGYNH5w4Gr5rCUbfK3Ds0bgnCsGrgXeBpYDLzrnlprZZDObHNzt98BhwBQzW2BmOV7VE7cqjhDuN14jhEViSNltpC/Ny6O0tNor5xFhzvn35g2RnZ3tcnLUL4BAIPzGrwJnBOMegEHKAkRizfUvLmT6l3mcP7QL90w40rP3MbN5zrns6p7TyOJYVG0gfIGagEgMemDiQACe+yKXvcUlvtSgRhBrygLhBf+CkTcoEBaJcUlJxp3j+wNw15vL/anBl3eV+qsuED75dwqEReJA2ZiCZz5bQ9H+yJ8VqBHEgvy8gwPhbsP9rkpEwiQlOYlzswMTK1z+dOQXuVcjiHZLZsCjxwdHCD+qEcIicer+iYM4PKspn363lfe/3hTR91YjiFaVAuEj4KrZCoRF4twzVxwLwGV/j+xZgZaqjEZ582D6FbB9dSAQHnWTsgCRBNCxZWNSkoziUkf+nv20aBKZ/+91RhBNSktg9oPw1ClQsl+BsEgCuu/swO2kZ035JGLvqUYQLcoD4Tuh7xnwMwXCIono7CGB0Hjllt0RG22sRhANlr5SJRD+OzRu5XdVIuKT284IrOF1yyuLI/J+agR+2rsTXr0GXrpUgbCIlPvJcd0AeH7u2tp3DBM1Ar+UjxD+Z3CE8Ntw2OF+VyUiUSApycrHFczP3e79+3n+DlKZAmERCcHFw7oBkVnBTI0gkvLz4B8/UiAsInUa0DGDVk1SeW3hOvIL93v6XmoEkVIeCC9QICwidTIzLglmBYNuf8fT91Ij8NreXQqERaRB/mdML45o0wyAt5d6t4qZGoGX8ubBYwqERaThHrt4CAD3zPRuimo1Ai9UDISL9ykQFpEGOzyrGSf1acPqrXv45fPzPXkPNYJwUyAsImE25cKjGXFEJv07ZHjy+pp0LpyWvgKv/xJKimH8FGUBIhIW6anJPHvlsZ69vhpBOOzdBW/dCAuehY5DYMLjygJEJGaoERyq7+fB9Cth2ypNGS0iMUmNoKFKS+CTP8H7d0OzdoFAWFmAiMQgNYKGyM+DGVfBmo+h/1lw+h81OExEYpYaQX0pEBaROKNGEKqKgXCHo+HsJxQIi0hcUCMIhQJhEYljagS1USAsIglAjaAmCoRFJEGoEVRHgbCIJBA1gooUCItIAlIjKFMpEL4eRt2sQFhEEoIawUGB8BvQbYTfVYmIRExiNwIFwiIiCdwIKgXCj8DgCxUIi0hC8nRhGjM71cy+NrMVZnZTNc+bmf0l+PwiMzvay3qAymsItz4cJn8ER12kJiAiCcuzMwIzSwYeAcYAecBcM3vNObeswm5jgZ7Bj2OBR4OfvaFAWETkIF5eGhoKrHDOrQQws+eB8UDFRjAeeMY554A5ZtbSzNo759aHvZqlr8L0K6BZWwXCIiIVeNkIOgJrKzzO4+C/9qvbpyNQqRGY2SRgUvDhLjP7uoE1ZcK2LVw/soHfHpMygS1+FxFhOubEoGOun641PeFlI6juortrwD4456YB0w65ILMc51z2ob5OLNExJwYdc2Lw6pi9DIvzgM4VHncC1jVgHxER8ZCXjWAu0NPMuptZGnAe8FqVfV4DLgnePTQMyPckHxARkRp5dmnIOVdsZtcCbwPJwFPOuaVmNjn4/FRgJjAOWAHsAS7zqp6gQ768FIN0zIlBx5wYPDlmC9ywIyIiicrTAWUiIhL91AhERBJcXDaCqJzawmMhHPOFwWNdZGafmtkgP+oMp7qOucJ+x5hZiZlNjGR9XgjlmM1slJktMLOlZvZhpGsMtxD+bbcws9fNbGHwmL3OGj1lZk+Z2SYzW1LD8+H//eWci6sPAsH0d0APIA1YCPSrss844C0C4xiGAZ/7XXcEjvl4oFXw67GJcMwV9vsvgRsTJvpddwR+zi0JjN7vEnzcxu+6I3DMtwD3Bb/OArYBaX7XfgjHfAJwNLCkhufD/vsrHs8Iyqe2cM7tA8qmtqiofGoL59wcoKWZtY90oWFU5zE75z51zm0PPpxDYMxGLAvl5wzwc2A6sCmSxXkklGO+AJjhnMsFcM7F+nGHcswOaG5mBjQj0AiKI1tm+DjnZhM4hpqE/fdXPDaCmqatqO8+saS+x3MFgb8oYlmdx2xmHYGzgKkRrMtLofycewGtzOwDM5tnZpdErDpvhHLMfwP6EhiMuhj4pXOuNDLl+SLsv7/icT2CsE1tEUNCPh4zG02gEcT6rHuhHPOfgBudcyUWH9OMh3LMKcAQ4GSgMfCZmc1xzn3jdXEeCeWYfwgsAE4CDgfeNbOPnHMFHtfml7D//orHRpCIU1uEdDxmNhB4AhjrnNsaodq8EsoxZwPPB5tAJjDOzIqdc69GpMLwC/Xf9hbn3G5gt5nNBgYBsdoIQjnmy4B7XeAC+gozWwX0Ab6ITIkRF/bfX/F4aSgRp7ao85jNrAswA7g4hv86rKjOY3bOdXfOdXPOdQNeBq6O4SYAof3b/jcw0sxSzKwJgRl/l0e4znAK5ZhzCZwBYWZtgd7AyohWGVlh//0Vd2cELjqntvBUiMf8e+AwYErwL+RiF8MzN4Z4zHEllGN2zi03s1nAIqAUeMI5V+1tiLEgxJ/zncDTZraYwGWTG51zMTs9tZk9B4wCMs0sD7gNSAXvfn9pigkRkQQXj5eGRESkHtQIREQSnBqBiEiCUyMQEUlwagQiIglOjUAkKDhD6QIzWxKczbJlmF9/tZllBr/eFc7XFjkUagQiBxQ65wY75wYQmPTrGr8LEokENQKR6n1GcCIvMzvczGYFJ3H7yMz6BLe3NbNXgvPgLzSz44PbXw3uu9TMJtX2JmbW3sxmVzgTGen5kYlUEXcji0UOlZklE5iy4MngpmnAZOfct2Z2LDCFwARnfwE+dM6dFfyeZsH9L3fObTOzxsBcM5tey9xOFwBvO+fuCr5GE6+OS6QmagQiBzQ2swVAN2AegVksmxFY1OelCjOYNgp+Pgm4BMA5VwLkB7f/wszOCn7dGegJ1NQI5gJPmVkq8KpzbkG4DkYkVLo0JHJAoXNuMNCVwGpY1xD4f2RHMDso++hb0wuY2SjgB8BxzrlBwHwgvab9g4uQnAB8D/xfHKwfIDFIjUCkCudcPvAL4AagEFhlZudA+XqxZes9vwf8LLg92cwygBbAdufcnmCWMKy29zKzrsAm59zjBC5Fxfz62RJ71AhEquGcm09gfdzzgAuBK8xsIbCUA0sl/hIYHZz1ch7QH5gFpJjZIgKzYs6p461GAQvMbD5wNvDnMB+KSJ00+6iISILTGYGISIJTIxARSXBqBCIiCU6NQEQkwakRiIgkODUCEZEEp0YgIpLg/h84CS0ElLH5kwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_recall_precision_curve(precisions,recalls):\n",
    "    x = [i/10000 for i in range(10001)]\n",
    "    plt.plot(recalls, precisions )\n",
    "    plt.plot(x,x)\n",
    "    plt.xlabel(\"Recalls\")\n",
    "    plt.ylabel(\"Precision\")\n",
    "    plt.ylim([0,1])\n",
    "plot_recall_precision_curve(precisions,recalls)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ROC 曲线"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "受试者工作特征（ROC）曲线是另一个二分类器常用的工具。它非常类似与准确率/召回率曲线，但不是画出准确率对召回率的曲线，ROC 曲线是真正例率（true positive rate，另一个名字叫做召回率）对假正例率（false positive rate, FPR）的曲线。FPR 是反例被错误分成正例的比率。它等于 1 减去真反例率（true negative rate， TNR）。TNR 是反例被正确分类的比率。TNR 也叫做特异性。所以 ROC 曲线画出召回率对（1 减特异性）的曲线。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了画出 ROC 曲线，你首先需要计算各种不同阈值下的 TPR、FPR，使用roc_curve()函数："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_curve\n",
    "fpr, tpr, thresholds = roc_curve(y_train_5, y_scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No handles with labels found to put in legend.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAA4XElEQVR4nO3dd3hUZfbA8e+ZSU9IgNBBioJIKKGKIiBFEayriLKKuDZEigqiiKjo8tPV1bU3EF1sa1kWBRURK6BYQEGagHQivZOEkGTm/P6YIUQMyRAyuTOT83meeTLlzr0nl3DPfd/33vOKqmKMMcYci8vpAIwxxoQ2SxTGGGOKZYnCGGNMsSxRGGOMKZYlCmOMMcWyRGGMMaZYQUsUIvKqiGwXkaXH+FxE5BkRWS0ii0WkbbBiMcYYU3rBbFFMBnoX83kfoIn/MQh4MYixGGOMKaWgJQpVnQPsLmaRS4DX1ed7oLKI1A5WPMYYY0onysFt1wU2FXqd4X9vy9ELisggfK0OEhMT25122mnlEqAxJvKoQr5XyfN4yPcoWuh9OPIahUMeL1EuQdX3+cE8D9FuQYE8j5esQx5iolwFyx+1BrTQ+x6v4lHFLXJkm2ih7wWHJ3M3nqw9oLpTVauXZh1OJgop4r0i95WqTgQmArRv314XLFgQzLiMMaWQk+cpOBh6vYpXfQdHryrZuR4yc/LJ93rxeJV8r3Iwz8O+7Dxiolx4VQuW9XrBo8rOzEOoQpRLyPcqufleft2ynzqV4wvWs2Z7FrHRLmKjXOR7fevYm53Hxt3ZvuU8XvI8XjbuzibK5SLX4z2u3+nwAVL8j8RCn8UCSWWz6wq4XUK1pBj2HcwjNTGWmsmxuF2CSwS3S8jzeDmQk8+pNSsR5RaiXEJslJsqCdFEu124/e/tysqlUWoCLpeLn+Z8xuIf5jDzvdc2lDYuJxNFBnBSodf1gM0OxWJMRMnzeNl3MI88j5e8fGV3di4er5e92XkczPMgCPleL15V8j3Khl3ZpMRHs3V/Dnuz84iLdvHThj2cVDWB3Hwv89fvplG1RDxeJc/jZc2OLGKjXMS4XeR5veTkHd8BuDzsO5j3h9eHk0S0W8jzKA1SE9iwK5tOp6RSJSEGl0twCbhFEBHcLnCJsGlPNi3qphDjduF2CXuz82hcI4lotyAIyfFR1KuSgIhved8D5KifLvGdGyfFRuFyiT8BUJAECn/vROzZs4dRo0bhPflkxo4dyxXtrweuR957rdTrdDJRTAeGicg7QEdgn6r+qdvJmEi2z3/gzjyUx4GcfDL2HCw4czyU72Xjrmyi3S7W7MikamIMh/K9LNy4hwapCQVn5vke5cd1uzm5eiK7s3LZk51LnqdsOjJWbD1Q8HzZ5v1/+OxQvi/GoyXFRvkOgC7BLVJwAM7J89IwNYEo/wE3yiXszsolPsZNnZR4RHxn1IUP1Nv2H+KkqvFUiosm2ncUJetQPg1TE3C7XES5hKzcfGqnxBET5Sp4z6tKSnw0cdFuolxCtNtFtaRY4qJdJ3wgDmXvv/8+Q4YMYceOHdx7771ltt6gJQoReRvoBlQTkQxgHBANoKovATOA84HVQDZwXbBiMSbYfF0euezMzGX7gRzyvcqhPC8/bdjN73sPkhQbxbLN+0mKjWL++t3UTonn970HS729wgfwot5zCXgVaiXHER0l7D+YT0yUi0bVEvl9z0FOrZlEYmwUUS4pOKBv3Z9Dq3opHMrzUjkhmuqVYsnzKHWrxBPjdiECVRJiiHIJUW4X0W7xHcDdQpTL9zqSD8KhbNu2bQwfPpz//ve/tG7dmo8//pi2bcvujoOgJQpV/WsJnyswtCy2lZeXR0ZGBjk5OX/6LC4ujnr16hEdHV0WmzIRKjffy47MQwV92nuz89i2/xDZufls25/DD+t2UyUhhh/W7aJ+1QSyDnlYvmU/tZLj2Lr/z393JSmcJKLdQs3kODL2HKRN/crszc6jeZ1kot2+vvvcfC8NUhNRlJOqJBDjdnEwz0OdyvFE+bswolxCTJSL1KRYaqfEERsV2WfO5o82bdrExx9/zEMPPcSdd95Z5sc7J7ueykxGRgaVKlWiYcOGf/jPoars2rWLjIwMGjVq5GCEJthUlQOH8lm7I4uDuR627PN14eR7fAOceV4vq7dnFgyert+VxcZd2SiwZd/xHei37T9U8LxwkoiLdlErOY7k+Gg27s6mQ8OqBf34J1dPpF6VBPI8voN+UmwUVRKjqVkpDpfLDujm+G3YsIEPP/yQYcOG0b59ezZu3EhqampQthURiSInJ+dPSQJ8g0Kpqans2LHDochMaagqe7Lz2Hcwjx0HDpHv9fL7noOo+rpXNu7Oxu2CXzbtK9XZfHFOqhpPtNvXz711Xw61UuLo2CiVelXiyfcqp1RPwiVQvVIs0W4XKfHRJMZGkRwXRZTbKuKY4PN6vbz44ovcfffdAPTt25fatWsHLUlAhCQKOPaVAtb8Dh2qyvpd2ezOOsT89XtYuyOT7QcOkZ3rYc32TFISosnJ9bD5OM/wj1Y1MYYWdVNYtzOT9g2q+vvUpeCKlVrJcTSpmYRLhFopcZxUJYGqSTEkx1n3pAltK1eu5MYbb+Sbb77hvPPOY8KECdSuHfz7lCMmUZjQsD8njy17c5i3ZiefLd9GSnw0363dRaW4KDbtLn7wdldW7p/eq5oYQ0p8NCdVTWD7/hw6nVKNqonRpCTEUD0phsTYKGqnxFOnchwJMfbnbCJXdnY2nTt3xuPxMHnyZAYOHFhuJ8L2P8scl33ZeXy3dicbdmXz88Y9bNx9EFUt8iqcwvZmH7mmPcoltK1fhb0Hc+nSpDq1U+KoXzWBOpXjSY6LJiUhmpR4O7s3BmDVqlU0adKEhIQE3njjDVq3bk2tWrXKNYaISRSqWmR29V1cZY5Hxp5sVm49wLLN+9mTncu6nVksztjH7iLO+IsSE+UChfqpCXQ6JZXuTWuQHB9F01rJJMVGzJ+cMUGVk5PD+PHjefTRR5k8eTIDBgygd+/i6qwGT0T8r42Li2PXrl2kpqYWedVTXFycg9GFrqW/7+OXjL3k5XtZvSOTaYs2cyAnv8TvpdVOJq1OMjWTY4mLcnNS1QRqpcRxcrVEqleKtXEhY07Qt99+yw033MDKlSu57rrruOCCCxyNJyISRb169cjIyCjy6qbD91FUZNm5+SzJ2MebP2zkp/W7Ax4sPqdZDWKj3PRsVoN6VRJIio2icY2kI0XQjDFlbvz48YwbN4769evz6aef0qtXL6dDioxEER0dbfdJ4GtB7T+Yz8JNe5i9age/bNrLzxv3FvudSnFRtKiTQlqdZJrVTqZjo6rUqRyP267tN6ZcHe4+b926NcOHD+ehhx4iKamsyw6WjoRbH75Vjz0i3+Plg0WbWZKxl+m/bGZPdl6xy7esm0LTWpU4p1kN2tavYt1ExoSA3bt3M2LECBo3bsx9990XtO2IyE+q2r40342IFkVFsTc7lxlLtjJ//W5+2bSXtTuzjrlslybVSI6L5sxTUulxWg3qVI4vx0iNMYGYMmUKQ4cOZffu3UFNEifKEkUI25+Tx+JN+3h3wSY+/OXYFdgrxUUx4IwGtKybwrlpNYm2O4SNCWlbtmxh2LBhTJ06lXbt2jFr1izS09OdDuuYLFGEkF2Zh/hq5Q4+WryZeat3FTnJSqW4KBqmJnJZ27qc1bgaTWokWfeRMWFm8+bNfPrppzz66KOMHDmSqKjQPhSHdnQVwLzVO/lg0e8sWL+nyK6k+lUTqJ0SR/M6KdzaszGVE2IciNIYc6LWr1/Phx9+yPDhw2nXrh2bNm2iSpUqTocVEEsU5UxV+XnjHj5evJUvVmxjw67sPy3TuXE1+p9+Ej1Oq2FlKYwJcx6Ph+eff5577rkHl8tFv379qFWrVtgkCbBEUW6278/h9e828M78jezMPHKHs0ugQ8OqNK+TwtDup5CaFOtglMaYsvTrr79y4403Mm/ePHr37s2ECRPKvfxGWbBEEWRLf9/H5S/N+9OcwqmJMQzv0ZiL0utYcjAmAmVnZ9O1a1e8Xi+vv/46AwYMCNvxREsUQbJ6eyZj31/CD+t2F7xXOyWOns1qcGuPJtRItrIixkSiFStW0LRpUxISEnjrrbdIT0+nZs2aTod1QixRlKE8j5c3v9/AtEWbWbRpb8H73ZtW594L0zilemjcZWmMKXsHDx7kgQce4PHHH+e1115jwIABIVF+oyxYoigD787fyP99/OufCuo1SE1gSLdTuLJDfYciM8aUhzlz5nDjjTfy22+/ceONN3LhhRc6HVKZskRxAvI9Xga98RNfrtj+h/cHntmAEeecSpVEu5TVmEj34IMP8sADD9CoUSM+//xzevbs6XRIZc4SRSnNX7+boW/9zPYDhwDf1Uv/uekMzjg5ePPWGmNCx+Eifu3bt2fEiBGMHz+exMREp8MKCisKeJwy9mTz7BereXfBJsA3W9tj/VpxaZuKXcrcmIpi586djBgxgiZNmnD//fc7HU7ATqQooBUFOg7/mrWSzo9+VZAkXAI/jj3HkoQxFYCq8t5775GWlsY777yDy1VxDp/W9RSAQ/kexkxdwtSffwegamIM/+zbinPSwvuSN2NMYDZv3syQIUOYNm0a7du35/PPP6dVq1ZOh1VuLFGU4JVv1jH+o+UFr//WqSH3X5iGyyb2MabC2Lp1K19++SWPPfYYt99+e8gX8StrFeu3PQ6qyv3TlvHG9xsK3nt5YHvOtVaEMRXC2rVrmT59Orfffjtt27Zl48aNVK5c2emwHGGJoggbdmVxzhOzyfP4BvqvOaMB4y5KI8rmeTAm4nk8Hp555hnGjh1LdHQ0/fv3p1atWhU2SYANZv/Jm99v4OzHvi5IEqN7n8b4v7SwJGFMBbBs2TLOOussRo4cSY8ePVi2bFlYFvEra9aiKGTinDU8PGNFwet//60D3U+r4WBExpjykp2dzdlnn42I8J///If+/fuHbRG/smaJwu/LFdsKksS1ZzbggYub2x+JMRXA8uXLadasGQkJCbzzzjukp6dTvXp1p8MKKRW+P0VVmThnDddP9t3Ed2Gr2pYkjKkAsrOzufPOO2nZsiVvvvkmAOecc44liSJU+BbFNa/8yDerdwLQ47QaPHlla0sSxkS4r7/+mptuuonVq1dz8803c/HFFzsdUkirsC2KzEP5DHz1SJI4rVYlXrm2PdE2aG1MRBs3bhzdu3dHVfnyyy956aWXSElJcTqskFZhj4oj313EnFU7ABjT5zRm3t7VWhLGRLDDde1OP/107rjjDhYvXkz37t0djio8BDVRiEhvEVkpIqtF5O4iPk8RkQ9F5BcRWSYi1wUznsNmLt3KrOXbAF+SuPnsU8pjs8YYB+zYsYOrrrqKv//97wBccMEFPP744yQkJDgcWfgIWqIQETfwPNAHSAP+KiJpRy02FFiuqulAN+BfIhLUSRy2789h+Ns/A3Bpm7qWJIyJUKrKf/7zH5o1a8aUKVOIibH5YUormC2K04HVqrpWVXOBd4BLjlpGgUri6/NJAnYD+QSJqnLRc9+Q51HqV01g/F9aBGtTxhgHZWRkcPHFF3P11VfTuHFjFi5cyJgxY5wOK2wFM1HUBTYVep3hf6+w54BmwGZgCXCbqnqPXpGIDBKRBSKyYMeOHaUO6N4PlrJtv2+ioZcGtCMptsJf9GVMRNqxYwdz5szhiSee4Ntvv6V58+ZOhxTWgpkoihoZPnqWpPOARUAdoDXwnIgk/+lLqhNVtb2qti/tNc7b9+fw358yABh3URppdf60GWNMGFu9ejVPPvkkAG3atGHTpk2MGDECt9vtcGThL5iJIgM4qdDrevhaDoVdB0xVn9XAOuC0YATz1Be/kZvvpVvT6lx3VqNgbMIY44D8/Hwef/xxWrZsyYMPPsi2bb4LVZKT7WSwrAQzUcwHmohII/8AdX9g+lHLbAR6AohITaApsLasA5m3eif/+WEjAMN7NC7r1RtjHLJkyRI6derEnXfeSa9evVi2bBk1a9pUAGUtaJ30qpovIsOATwE38KqqLhORwf7PXwLGA5NFZAm+rqrRqrqzjOPgrv8tBqBdgyq0a1C1LFdvjHFIdnY23bt3x+Vy8c4773DFFVfYvVBBEtTRXFWdAcw46r2XCj3fDPQKZgxvfr+BjD0HAXjh6rbB3JQxphwsXbqU5s2bk5CQwLvvvkt6ejrVqlVzOqyIFtF3Zqsqb/m7nC5oVZuayXEOR2SMKa2srCxGjhxJq1atCor49ezZ05JEOYjo60M/WryFFVsPUDkhmieuSHc6HGNMKX3xxRfcdNNNrFu3jiFDhnDJJUffkmWCKaJbFBPn+MbFb+pyMrFRdomcMeHovvvu45xzziEqKorZs2fz/PPP2xVN5SxiE8UHC39nye/7iIt2cd1ZDZ0OxxhznLxe3723nTp14q677uKXX36ha9euDkdVMUVsovjMX/SvRZ0UEmIiuofNmIiyfft2+vfvz4MPPghAnz59ePTRR4mPj3c4soorIhOFx6t8vGQLALf2bOJwNMaYQKgqb775Js2aNeP999+36q4hJCITxdSffaU6ot3CWY3tighjQt2mTZu48MILueaaa2jatCkLFy5k9OjRTodl/CIyUbz67XoAru7YALfLbsAxJtTt2rWLb7/9lqeffpq5c+eSlnb0jATGSRHXeb9570F+3bIfgME214QxIWvVqlVMnz6dUaNG0bp1azZt2kSlSpWcDssUIeJaFJ8s3VrwvFaK3WBnTKjJz8/n0UcfpVWrVjz00EMFRfwsSYSuiEsUz335GwB/v8TqzxsTan755Rc6duzI3Xffzfnnn8/y5cutiF8YiKiup3U7s9iTnQdA37b1HI7GGFNYdnY2PXv2JCoqiilTptC3b1+nQzIBiqhE8ekyX7dTx0ZVSbTZ64wJCYsXL6Zly5YkJCTw3//+l/T0dKpWtSrO4SSiup7e/tFXAPCK9ieVsKQxJtgyMzO57bbbaN26NW+88QYA3bt3tyQRhiLmtNvrVbb758M+85RUh6MxpmL77LPPGDRoEOvXr2fYsGFceumlTodkTkDEtCiW/L6Pg3keRKBOZbvV3xinjB07ll69ehEbG8vcuXN59tln7YqmMBdwohCRxGAGcqIWbNgDwIWt6jgciTEV0+Eifp07d2bMmDEsWrSIzp07OxyVKQslJgoR6SQiy4Ff/a/TReSFoEd2nOb+tgOAs6zbyZhytXXrVi6//HIeeOABwFfE7+GHHyYuzu5jihSBtCieBM4DdgGo6i9AyNX6XbRpLwBnnGyJwpjyoKpMnjyZtLQ0PvroI5sjIoIFNJitqpuOmrTcE5xwSmfLvoPszc7D7RIapFrFSWOCbcOGDQwaNIhZs2bRuXNnJk2aRNOmTZ0OywRJIC2KTSLSCVARiRGRUfi7oULF7JW+bqdG1RI5KqEZY4Jg7969zJ8/n+eee47Zs2dbkohwgbQoBgNPA3WBDGAWMCSYQR2vXzL2AtAwNaTH240JaytXrmT69OnceeedpKens3HjRpKSkpwOy5SDQFoUTVX1alWtqao1VHUA0CzYgQVKVQtmsxva3arFGlPW8vLy+Mc//kF6ejqPPPII27dvB7AkUYEEkiieDfA9R2TsOcjOzFwSY9y0Pqmy0+EYE1EWLlxIx44dueeee7joootYvnw5NWrUcDosU86O2fUkImcCnYDqIjKy0EfJgDvYgQVq5dYDAJxaq5KNTxhThrKzszn33HOJjo7mf//7H5dddpnTIRmHFDdGEQMk+ZcpfFvlfuDyYAZ1PFZu8yWKuKiQyV3GhLWFCxfSunVrEhISmDJlCunp6VSpUsXpsIyDjpkoVHU2MFtEJqvqhnKM6bis3ZEFQLsG9odszIk4cOAAY8aM4fnnn+e1115j4MCBdOvWzemwTAgI5KqnbBF5DGgOFNxqqao9ghbVcfhx/S4AatpsdsaU2syZM7n55pvZtGkTt912m3UzmT8IZDD7LWAF0Ah4EFgPzA9iTMflYK7v3r9OVrrDmFIZM2YMffr0ITExkW+//ZannnrKrmgyfxBIiyJVVV8RkdsKdUfNDnZggdi2P4edmblEu4WTq9k9FMYcD4/Hg9vtplu3bkRFRXHvvfcSGxvrdFgmBAWSKPL8P7eIyAXAZiAk5hk9PD7Rsm6KXfFkTIC2bNnC0KFDad68OePHj+e8887jvPPOczosE8IC6Xr6PxFJAe4ARgGTgNuDGVSgft97EIC6Vay+kzElUVX+/e9/k5aWxieffGJXMpmAldiiUNWP/E/3Ad0BROSsYAYVqN/3+BJFHRvINqZY69ev56abbuLzzz+nS5cuTJo0iVNPPdXpsEyYKO6GOzdwBb4aTzNVdamIXAjcA8QDbconxGNb5b+Hol4Vm9HOmOLs27ePn3/+mRdeeIGbb74ZlytiJrc05aC4FsUrwEnAj8AzIrIBOBO4W1U/CGTlItIbX0FBNzBJVR8pYpluwFNANLBTVc8ONPjPfvXVeKpeyVoUxhxt+fLlTJ8+nbvvvrugiF9iol30YY5fcYmiPdBKVb0iEgfsBBqr6tZAVuxvkTwPnIuv6ux8EZmuqssLLVMZeAHoraobReS4ishUSYhm2/5D1LU5so0pkJubyz//+U/Gjx9PpUqVuP7666lRo4YlCVNqxbU/c1XVC6CqOcCqQJOE3+nAalVdq6q5wDvAJUctcxUwVVU3+rezPdCVe73Ktv2HAGhS0675NgZgwYIFdOjQgfvuu4/LLrvMiviZMlFci+I0EVnsfy7AKf7XAqiqtiph3XWBTYVeZwAdj1rmVCBaRL7GV0/qaVV9/egVicggYBBA/fr1Adh+wJck4qJdxEVbnSdjsrKyOO+884iLi2PatGlcfPHFTodkIkRxieJE55wo6sYGLWL77YCe+AbIvxOR71V11R++pDoRmAjQvn17BVixdT8AqYl2g5Cp2H7++Wdat25NYmIi77//Pq1ataJy5cpOh2UiyDG7nlR1Q3GPANadgW8w/LB6+G7WO3qZmaqapao7gTlAeiCBr9/pu9nuTCvdYSqo/fv3M2TIENq1a8ebb74JQNeuXS1JmDIXzGvk5gNNRKSRiMQA/YHpRy0zDegiIlEikoCvayqg+bi/WOEbzoi3bidTAc2YMYPmzZszYcIERo4cSd++fZ0OyUSwoCUKVc0HhgGf4jv4v6eqy0RksIgM9i/zKzATWIzvMtxJqro0kPUfvnciJ88ThOiNCV2jR4/mggsuIDk5mXnz5vGvf/3LrmgyQRVIrSdEJB6or6orj2flqjoDmHHUey8d9fox4LHjWS/AnixfCaqzm1Y/3q8aE3ZUFa/Xi9vtpmfPnsTFxXHPPfdYET9TLkpsUYjIRcAifGf+iEhrETm6C6ncbdnnK99Rw262MxHu999/5y9/+Qvjxo0DoFevXjz44IOWJEy5CaTr6QF890TsBVDVRUDDYAUUqM37cgCobXWeTIRSVV5++WXS0tKYNWsW1apVczokU0EF0vWUr6r7QqmMt6qyNzsXgOqV7KzKRJ5169Zxww038NVXX9GtWzdefvllGjdu7HRYpoIKJFEsFZGrALeINAFuBeYFN6ziZed6yPMo8dFuu9nORKTMzEwWL17MhAkTuPHGG62In3FUIH99w/HNl30I+A++cuO3BzGmEu3xtybyvV4nwzCmTC1dupSHH34YgJYtW7Jx40YGDRpkScI4LpC/wKaqOlZVO/gf9/prPzkm81A+AHFR1pow4S83N5cHH3yQtm3b8uSTT7J9u+8eoYQEm5DLhIZAEsUTIrJCRMaLSPOgRxSAnQd8LYrUpBiHIzHmxMyfP5927drxwAMP0K9fPyviZ0JSIDPcdReRWvgmMZooIsnAu6r6f0GP7hj25/juoUhNsoFsE76ysrLo3bs38fHxTJ8+nYsuusjpkIwpUkCdn6q6VVWfAQbju6fi/mAGVZJNu7MBaFk3xckwjCmVBQsW4PV6SUxMZNq0aSxbtsyShAlpgdxw10xEHhCRpcBz+K54qhf0yIqx2y6NNWFo37593HzzzXTo0KGgiF/nzp1JSbETHhPaArk89t/A20AvVT26+qsjlmTsAyApNqAKJMY47sMPP2Tw4MFs3bqVUaNGcfnllzsdkjEBC2SM4ozyCOR4uPw3/+Xm2+WxJvTdeeedPP7447Rs2ZIPPviADh06OB2SMcflmIlCRN5T1StEZAl/nHAo0Bnugmbrft/VufVT7fJBE5pUFY/HQ1RUFL169SI5OZnRo0cTE2NX6pnwU1yL4jb/zwvLI5DjkRjju3+iaqL9pzOhJyMjg1tuuYVWrVrx0EMPce6553Luuec6HZYxpVbcDHdb/E+HFDG73ZDyCa9oh2+4S4mPdjIMY/7A6/UyYcIE0tLS+PLLL6lVq5bTIRlTJgK5PLaoU6E+ZR3I8dh30JcokuMsUZjQsHbtWnr06MHgwYM5/fTTWbJkCcOHD3c6LGPKRHFjFLfgazmcLCKLC31UCfg22IEV54D/hrvkeLvqyYSGrKwsli9fzqRJk7j++usJpWrLxpyo4o60/wE+Af4B3F3o/QOqujuoURXDq3Ao30uM22XzZRtHLVmyhGnTpnHvvffSsmVLNmzYQHx8vNNhGVPmiut6UlVdDwwFDhR6ICJVgx9a0bxe3wVYyfFRdtZmHHHo0CHuv/9+2rZtyzPPPFNQxM+ShIlUJbUoLgR+wnd5bOGjsgInBzGuY/KoL1HYzXbGCd9//z033HADy5cv55prruHJJ58kNTXV6bCMCapjHm1V9UL/z0blF07J1J8obMIiU96ysrK44IILSExMZMaMGfTp4+g1HcaUm0BqPZ0lIon+5wNE5AkRqR/80IqW7+96irVEYcrJDz/8UFDE78MPP2TZsmWWJEyFEsjlsS8C2SKSDtwFbADeCGpUxfA3KNi4K8upEEwFsXfvXm688UbOOOOMgiJ+nTp1olKlSg5HZkz5CiRR5Kuvv+cS4GlVfRrfJbKO8PozRYeGjo2nmwrggw8+IC0tjcmTJzN69Gj69evndEjGOCaQEeEDIjIGuAboIiJuwLE73Q4niqQ4G8w2wTFy5EiefPJJ0tPT+fDDD2nXrp3TIRnjqECOtlcCVwHXq+pW//jEY8EN69i8/oKxiTGWKEzZKVzE7/zzzyc1NZW77rqL6Gi7+9+YErueVHUr8BaQIiIXAjmq+nrQIztWPP5CtjFRAU3OZ0yJNm7cyAUXXMC4ceMAOOeccxg7dqwlCWP8Arnq6QrgR6AfvnmzfxARx2ZdycnzNSliLVGYE+T1ennhhRdo3rw5s2fPpk6dOk6HZExICqT/ZizQQVW3A4hIdeBzYEowAzuWKJfvvr89/ulQjSmN1atXc/311zN37lzOPfdcJk6cSMOGDZ0Oy5iQFEiicB1OEn67COxqqaA4mOehEtCsdrJTIZgIkJOTw6pVq/j3v//Ntddea+VgjClGIIlipoh8im/ebPANbs8IXkjFc/tbFAdzPU6FYMLUokWLmDZtGuPGjaNFixasX7+euLg4p8MyJuQFMph9JzABaAWkAxNVdXSwAzt2PL6fJ1W1aVBNYHJychg7dizt27fnxRdfLCjiZ0nCmMAUNx9FE+Bx4BRgCTBKVX8vr8COJd9rg9kmcPPmzeOGG25gxYoVXHvttTzxxBNUrWo3axpzPIrrenoVeB2YA1wEPAtcVh5BFSff42tSWJeyKUlWVhYXXXQRSUlJzJw5k/POO8/pkIwJS8Ulikqq+rL/+UoR+bk8AipJlNuXIeKj7YY7U7TvvvuOjh07kpiYyEcffUSLFi2sPpMxJ6C4/ps4EWkjIm1FpC0Qf9TrEolIbxFZKSKrReTuYpbrICKeQO7PODxGkRhr1WPNH+3Zs4frr7+eTp068cYbvrqVZ555piUJY05QcaflW4AnCr3eWui1Aj2KW7G/JtTzwLlABjBfRKar6vIilnsU+DSQgA/l+8Yo7M5sU9jUqVMZOnQoO3bsYMyYMVx55ZVOh2RMxChu4qLuJ7ju04HVqroWQETewVeBdvlRyw0H/gd0OJ6VH77xzpgRI0bw1FNP0bp1a2bMmEGbNm2cDsmYiBLMjv66wKZCrzOAjoUXEJG6wKX4WifHTBQiMggYBBBX6xQAEm0q1AqtcBG/Cy+8kBo1ajBq1Cirz2RMEASz/6aoU3496vVTwGhVLfbuOVWdqKrtVbW9uHwhx7it66miWr9+Pb179+a+++4DoGfPnowZM8aShDFBEsyjbQZwUqHX9YDNRy3THnhHRNYDlwMviMhfilupx2vVYysqr9fLs88+S4sWLZg3bx4NGjRwOiRjKoQS+2/EVwTnauBkVf27fz6KWqr6YwlfnQ80EZFGwO9Af3zzWhRQ1UaFtjMZ+EhVPwgk8Ngou+qpIvntt9+47rrr+Pbbb+nduzcvvfSSJQpjykkgp+UvAGcCf/W/PoDvaqZiqWo+MAzf1Uy/Au+p6jIRGSwig0sZbwFrUVQsubm5rFmzhtdff50ZM2ZYkjCmHAUyItxRVduKyEIAVd0jIjGBrFxVZ3BUAUFVfekYy/4tkHUCJMRYa6IiWLhwIdOmTeOBBx6gefPmrF+/ntjYWKfDMqbCCeS0PM9/r4NCwXwU3qBGVQJrTUS2nJwcxowZQ4cOHZgwYQI7duwAsCRhjEMCOeI+A7wP1BCRh4BvgIeDGlUJbL7syPXNN9+Qnp7OI488wsCBA1m+fDnVq1d3OixjKrQSj7iq+paI/AT0xHfJ619U9degR1YMa1FEpszMTC655BKSk5OZNWsW5557rtMhGWMI7Kqn+kA28GHh91R1YzADK47dlR1ZvvnmGzp16kRSUhIff/wxLVq0ICkpyemwjDF+gZyafwx85P/5BbAW+CSYQZUk2m62iwi7du1i4MCBdOnSpaCI3xlnnGFJwpgQE0jXU8vCr/2VY28OWkQB2Hcwz8nNmxOkqkyZMoVhw4axe/du7rvvPvr37+90WMaYYzjuUWFV/VlEjquAX1mzMYrwNmLECJ5++mnatWvHrFmzSE9PdzokY0wxAhmjGFnopQtoC+wIWkQBqFcl3snNm1JQVfLz84mOjubiiy+mTp06jBw5kqgou4LNmFAXyKl5pUKPWHxjFZcEM6iS2GB2eFm3bh29evUqKOLXo0cP7rrrLksSxoSJYv+n+m+0S1LVO8spnoDYYHZ48Hg8PPfcc9xzzz243W769evndEjGmFI4ZqIQkShVzQ902tPytGZHptMhmBKsWrWKv/3tb3z33Xf06dOHCRMmcNJJJ5X8RWNMyCmuRfEjvvGIRSIyHfgvkHX4Q1WdGuTYjql5nRSnNm0ClJ+fz4YNG3jzzTe56qqr8BUhNsaEo0A6iasCu/DNQqf47s5WwLFE4bYxipC0YMECpk2bxvjx40lLS2Pt2rVWn8mYCFBcZ38N/xVPS4El/p/L/D+XlkNsx2SJIrQcPHiQu+66i44dO/Lqq69aET9jIkxxicINJPkflQo9P/xwjNu6MULG7NmzadWqFY899hg33HADy5YtsyJ+xkSY4rqetqjq38stkuPgshZFSMjMzOSyyy6jcuXKfPHFF/To0cPpkIwxQVBcogjZo7HdR+GsuXPnctZZZ5GUlMQnn3xC8+bNSUxMdDosY0yQFNf11LPcojhONkbhjJ07dzJgwAC6du1aUMTv9NNPtyRhTIQ7ZotCVXeXZyDHw6vqdAgViqry3nvvMXz4cPbs2cO4ceOsiJ8xFUhY1lDYvv+Q0yFUKLfddhvPPvssHTp04IsvvqBly5Ylf8kYEzHCMlE0rGZdHcGmquTl5RETE8Oll15KgwYNuP3223G73U6HZowpZ2FZNMmGKIJrzZo19OzZk3vvvReA7t27c8cdd1iSMKaCCstEYYPZweHxeHjiiSdo2bIlP/30E02bNnU6JGNMCAjLrierG1T2VqxYwbXXXsuPP/7IRRddxIsvvkjdunWdDssYEwLCMlHYndllz+v1snnzZt5++22uvPJKS8bGmAJhmSis56ls/Pjjj0ybNo2HHnqItLQ01qxZQ0xMjNNhGWNCTFiOUVgJjxOTnZ3NqFGjOPPMM3nttdcKivhZkjDGFCU8E4V1i5TaV199RcuWLfnXv/7FTTfdZEX8jDElCsuuJ5sJtXQyMzPp168flStX5quvvqJbt25Oh2SMCQNheci1FsXx+frrr/F6vQVF/BYvXmxJwhgTsLBMFHYfRWB27NjBX//6V7p3786bb74JQIcOHUhISHA4MmNMOAnLricrM148VeXtt9/m1ltv5cCBA4wfP96K+BljSi0sEwXW9VSs4cOH8/zzz3PGGWfwyiuvkJaW5nRIxpgwFpaJwhoUf+b1esnPzycmJobLL7+cxo0bM3z4cKvPZIw5YUEdoxCR3iKyUkRWi8jdRXx+tYgs9j/miUh6IOu1wew/+u233+jRowdjx44FoFu3blbp1RhTZoKWKETEDTwP9AHSgL+KyNF9IOuAs1W1FTAemBjIuq1F4ZOfn8/jjz9Oq1atWLRoEc2aNXM6JGNMBApm19PpwGpVXQsgIu8AlwDLDy+gqvMKLf89UC+QFVsdIvj1118ZOHAgCxYs4JJLLuGFF16gTp06TodljIlAwex6qgtsKvQ6w//esdwAfFLUByIySEQWiMgCsK6nw7Zt28a7777L+++/b0nCGBM0wWxRFHU0L3KyaxHpji9RdC7qc1WdiL9bKrZ2E62oaeL7779n2rRp/OMf/6BZs2asWbOG6Ohop8MyxkS4YLYoMoCTCr2uB2w+eiERaQVMAi5R1V2BrNgVlrcJll5WVhYjRoygU6dOvPXWWwVF/CxJGGPKQzAPufOBJiLSSERigP7A9MILiEh9YCpwjaquCnTFFanr6fPPP6dFixY89dRTDBkyxIr4GWPKXdC6nlQ1X0SGAZ8CbuBVVV0mIoP9n78E3A+kAi/4B6jzVbV9SeuuKIPZmZmZ9O/fn6pVqzJnzhy6dOnidEjGmApIVIscNghZsbWb6P8+nc2FrSJ38PbLL7/k7LPPxu1289NPP5GWlkZ8fLzTYRljwpiI/BTIiXhRwrK3P1K7nrZt28YVV1xBz549C4r4tWvXzpKEMcZRYZkoIi1NqCpvvPEGaWlpBVOTXnXVVU6HZYwxQJjWesrO9TgdQpkaOnQoL774ImeeeSavvPKK3WFtjAkpYZkoEmLCv4aR1+slLy+P2NhYrrzySpo1a8aQIUOsPpMxJuSEZddTdJjPhbpy5UrOPvvsgiJ+Z599tlV6NcaErPA+4oaZvLw8HnnkEdLT01m6dCktW7Z0OiRjjClRWHY9heNFT8uWLeOaa65h4cKFXHbZZTz//PPUqlXL6bCMMaZElijKidvtZvfu3UyZMoW+ffs6HY4xxgQsLLueJEwukJ03bx6jR48G4LTTTmP16tWWJIwxYScsE0Wo54nMzExuvfVWOnfuzLvvvsvOnTsBiIoKywacMaaCC89EEcJmzZpFixYteO655xg2bBhLly6lWrVqTodljDGlFpanuKHaoMjMzOTqq68mNTWVuXPnctZZZzkdkjHGnLCwbFGEWvXYzz77DI/HQ1JSErNmzWLRokWWJIwxESM8E4XTAfht2bKFvn370qtXL9566y0A2rRpQ1xcnMORGWNM2QnPROFwplBVJk+eTFpaGh9//DGPPPKIFfEzxkSssByjcNott9zChAkT6Ny5M5MmTaJp06ZOh2SMMUETlonCifsoChfxu+qqq2jVqhWDBw/GVdEm8DbGVDhheZQr766nX3/9lS5dunDPPfcA0LVrV4YMGWJJwhhTIYTlka688kReXh4PP/wwrVu3ZsWKFbRp06actmyMMaEjLLueysOyZcsYMGAAixYtol+/fjz77LPUrFnT6bCMMabchWeiKIcmRVRUFPv27WPq1Klceumlwd+gMcaEqDDtegpOppg7dy6jRo0CoGnTpqxatcqShDGmwgvPRFHGeeLAgQMMHTqUrl27MnXqVCviZ4wxhYRnoijDdX3yySc0b96cF198kdtvv50lS5ZYET9jjCmkQp8yHzhwgIEDB1KjRg3mzZvHGWec4XRIxhgTcsKzRXECfU+qysyZM/F4PFSqVInPP/+cn3/+2ZKEMcYcQ5gmitJ9b8uWLVx22WX06dOnoIhfeno6sbGxZRidMcZElvBMFMe5vKry6quv0qxZM2bOnMk///lPK+JnjDEBCssxiuNtUQwePJiJEyfStWtXJk2aRJMmTYITmDHGRKCwTBSB8Hg85OXlERcXx4ABA2jTpg2DBg2y+kzGGHOcwvSoWXyTYtmyZZx11lkFRfy6dOlilV6NMaaUwvLIeayup9zcXMaPH0+bNm1YvXo1HTp0KN/AjDEmAoVl11NReWLJkiVcffXVLFmyhP79+/PMM89QvXr1co/NGGMiTXgmiiKaFDExMWRnZzNt2jQuvvhiB6IyxpjIFJZdT4fNnj2bO+64A/AV8Vu5cqUlCWOMKWNBTRQi0ltEVorIahG5u4jPRUSe8X++WETaBrLerAP7ueWWW+jWrRsffPBBQRE/t9tdxr+BMcaYoCUKEXEDzwN9gDTgryKSdtRifYAm/scg4MWS1us9lEXfc85k4sSJjBw50or4GWNMkAVzjOJ0YLWqrgUQkXeAS4DlhZa5BHhdVRX4XkQqi0htVd1yrJXm791GUpOmTHt/Kh07dgxi+MYYYyC4iaIusKnQ6wzg6CN7UcvUBf6QKERkEL4WB8ChNat+XWpF/ACoBux0OogQYfviCNsXR9i+OKJpab8YzERR1FWsWoplUNWJwEQAEVmgqu1PPLzwZ/viCNsXR9i+OML2xREisqC03w3mYHYGcFKh1/WAzaVYxhhjjIOCmSjmA01EpJGIxAD9gelHLTMdGOi/+ukMYF9x4xPGGGPKX9C6nlQ1X0SGAZ8CbuBVVV0mIoP9n78EzADOB1YD2cB1Aax6YpBCDke2L46wfXGE7YsjbF8cUep9Ib4LjowxxpiihfWd2cYYY4LPEoUxxphihWyiCFb5j3AUwL642r8PFovIPBFJdyLO8lDSvii0XAcR8YjI5eUZX3kKZF+ISDcRWSQiy0RkdnnHWF4C+D+SIiIfisgv/n0RyHho2BGRV0Vku4gsPcbnpTtuqmrIPfANfq8BTgZigF+AtKOWOR/4BN+9GGcAPzgdt4P7ohNQxf+8T0XeF4WW+xLfxRKXOx23g38XlfFVQqjvf13D6bgd3Bf3AI/6n1cHdgMxTscehH3RFWgLLD3G56U6boZqi6Kg/Ieq5gKHy38UVlD+Q1W/ByqLSO3yDrQclLgvVHWequ7xv/we3/0okSiQvwuA4cD/gO3lGVw5C2RfXAVMVdWNAKoaqfsjkH2hQCXxzVGQhC9R5JdvmMGnqnPw/W7HUqrjZqgmimOV9jjeZSLB8f6eN+A7Y4hEJe4LEakLXAq8VI5xOSGQv4tTgSoi8rWI/CQiA8stuvIVyL54DmiG74beJcBtquotn/BCSqmOm6E6cVGZlf+IAAH/niLSHV+i6BzUiJwTyL54Chitqp6iJriKIIHsiyigHdATiAe+E5HvVXVVsIMrZ4Hsi/OARUAP4BTgMxGZq6r7gxxbqCnVcTNUE4WV/zgioN9TRFoBk4A+qrqrnGIrb4Hsi/bAO/4kUQ04X0TyVfWDcomw/AT6f2SnqmYBWSIyB0gHIi1RBLIvrgMeUV9H/WoRWQecBvxYPiGGjFIdN0O168nKfxxR4r4QkfrAVOCaCDxbLKzEfaGqjVS1oao2BKYAQyIwSUBg/0emAV1EJEpEEvBVb/61nOMsD4Hsi434WlaISE18lVTXlmuUoaFUx82QbFFo8Mp/hJ0A98X9QCrwgv9MOl8jsGJmgPuiQghkX6jqryIyE1gMeIFJqlrkZZPhLMC/i/HAZBFZgq/7ZbSqRlz5cRF5G+gGVBORDGAcEA0ndty0Eh7GGGOKFapdT8YYY0KEJQpjjDHFskRhjDGmWJYojDHGFMsShTHGmGJZojAhyV/5dVGhR8Nils0sg+1NFpF1/m39LCJnlmIdk0Qkzf/8nqM+m3eiMfrXc3i/LPVXQ61cwvKtReT8sti2qbjs8lgTkkQkU1WTynrZYtYxGfhIVaeISC/gcVVtdQLrO+GYSlqviLwGrFLVh4pZ/m9Ae1UdVtaxmIrDWhQmLIhIkoh84T/bXyIif6oaKyK1RWROoTPuLv73e4nId/7v/ldESjqAzwEa+7870r+upSJyu/+9RBH52D+3wVIRudL//tci0l5EHgHi/XG85f8s0//z3cJn+P6WTF8RcYvIYyIyX3zzBNwcwG75Dn9BNxE5XXxzkSz0/2zqv0v578CV/liu9Mf+qn87C4vaj8b8idP10+1hj6IegAdfEbdFwPv4qggk+z+rhu/O0sMt4kz/zzuAsf7nbqCSf9k5QKL//dHA/UVsbzL+uSuAfsAP+ArqLQES8ZWmXga0AfoCLxf6bor/59f4zt4LYiq0zOEYLwVe8z+PwVfJMx4YBNzrfz8WWAA0KiLOzEK/33+B3v7XyUCU//k5wP/8z/8GPFfo+w8DA/zPK+Or+5To9L+3PUL7EZIlPIwBDqpq68MvRCQaeFhEuuIrR1EXqAlsLfSd+cCr/mU/UNVFInI2kAZ86y9vEoPvTLwoj4nIvcAOfFV4ewLvq6+oHiIyFegCzAQeF5FH8XVXzT2O3+sT4BkRiQV6A3NU9aC/u6uVHJmRLwVoAqw76vvxIrIIaAj8BHxWaPnXRKQJvmqg0cfYfi/gYhEZ5X8dB9QnMmtAmTJiicKEi6vxzUzWTlXzRGQ9voNcAVWd408kFwBviMhjwB7gM1X9awDbuFNVpxx+ISLnFLWQqq4SkXb4aub8Q0RmqerfA/klVDVHRL7GV/b6SuDtw5sDhqvqpyWs4qCqthaRFOAjYCjwDL5aRl+p6qX+gf+vj/F9Afqq6spA4jUGbIzChI8UYLs/SXQHGhy9gIg08C/zMvAKvikhvwfOEpHDYw4JInJqgNucA/zF/51EfN1Gc0WkDpCtqm8Cj/u3c7Q8f8umKO/gK8bWBV8hO/w/bzn8HRE51b/NIqnqPuBWYJT/OynA7/6P/1Zo0QP4uuAO+xQYLv7mlYi0OdY2jDnMEoUJF28B7UVkAb7WxYoilukGLBKRhfjGEZ5W1R34Dpxvi8hifInjtEA2qKo/4xu7+BHfmMUkVV0ItAR+9HcBjQX+r4ivTwQWHx7MPsosfHMbf66+qTvBN5fIcuBnEVkKTKCEFr8/ll/wldX+J77Wzbf4xi8O+wpIOzyYja/lEe2Pban/tTHFsstjjTHGFMtaFMYYY4plicIYY0yxLFEYY4wpliUKY4wxxbJEYYwxpliWKIwxxhTLEoUxxphi/T9FwppcX4f5LAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def plot_roc_curve(fpr, tpr, label=None):\n",
    "    plt.plot(fpr, tpr, linewidth=2, label=label)\n",
    "    plt.plot([0, 1], [0, 1], 'k--')\n",
    "    plt.axis([0, 1, 0, 1])\n",
    "    plt.xlabel('False Positive Rate')\n",
    "    plt.ylabel('True Positive Rate')\n",
    "    plt.legend(loc = \"best\")\n",
    "plot_roc_curve(fpr, tpr)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里同样存在折衷的问题：召回率（TPR）越高，分类器就会产生越多的假正例（FPR）。图中的点线是一个完全随机的分类器生成的 ROC 曲线；一个好的分类器的 ROC 曲线应该尽可能远离这条线（即向左上角方向靠拢）。\n",
    "\n",
    "一个比较分类器之间优劣的方法是：测量 ROC 曲线下的面积（AUC）。一个完美的分类器的 ROC AUC 等于 1，而一个纯随机分类器的 ROC AUC 等于 0.5。Scikit-Learn 提供了一个函数来计算 ROC AUC："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9463859969616196"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "roc_auc_score(y_train_5,y_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "让我们训练一个RandomForestClassifier，然后拿它的的 ROC 曲线和 ROC AUC 数值去跟SGDClassifier的比较。首先你需要得到训练集每个样例的数值。但是由于随机森林分类器的工作方式，RandomForestClassifier不提供decision_function()方法。相反，它提供了predict_proba()方法。Skikit-Learn 分类器通常二者中的一个。predict_proba()方法返回一个数组，数组的每一行代表一个样例，每一列代表一个类。数组当中的值的意思是：给定一个样例属于给定类的概率。比如，70%的概率这幅图是数字 5。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "forest_clf = RandomForestClassifier(random_state = 42)\n",
    "y_proba_forest = cross_val_predict(forest_clf, X_train, y_train_5, cv=3,\n",
    "                                  method = \"predict_proba\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "但是要画 ROC 曲线，你需要的是样例的分数，而不是概率。一个简单的解决方法是使用正例的概率当作样例的分数。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_scores_forest = y_proba_forest[:,1]# score = proba of positive class\n",
    "fpr_forest,tpr_forest,thresholds_forest = roc_curve(y_train_5,y_scores_forest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在你即将得到 ROC 曲线。将前面一个分类器的 ROC 曲线一并画出来是很有用的，可以清楚地进行比较。见图 3-7。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAABADUlEQVR4nO3deXgUVdbA4d/p7uwhgRAihF1FZDVsg4h+iriAbK6g4ogL4wrK4IbbDDo6OooLiIhBRR0RVERAZBkFERGUNeygCAFCiFkghBBC0un7/dFNjBiSTkinupPzPk8/6equrjopQp2+91adK8YYlFJKqVOxWR2AUkop/6aJQimlVJk0USillCqTJgqllFJl0kShlFKqTJoolFJKlclniUJE3hORdBHZfIr3RUQmiMhOEdkoIp19FYtSSqnK82WL4n2gTxnv9wVaeR53AW/5MBallFKV5LNEYYxZBhwsY5VBwIfG7Uegrog08lU8SimlKsdh4b4bA/tKLKd4Xjtw8ooichfuVgcRERFdzj333IrvzbjA5fT8LAJXIRR5Hi6n5zUnmCL3Osb88Sd6B7tSKvAcyDWk5RpchkxjTIPKbMPKRCGlvFbq2dgYkwgkAnTt2tWsWbOm7C0f3AX7VkPmDjiUDGmbIXPnqTZ/ErvnUUq4jlBwBLt/2oNBbO6HzQ5i//2niPt9ezDYg8p+brO7ty0lDocIxYenrOdi83zWdtKylPN+yWV+Xz45jpP/iU71npz8T1mZ90r7c1CBpKDIhcsYjDG4jPs7lnsZjhe5OF5QRJExuDzvFzhd5BU4cdhtns8Yz2fcn8vNd2IAu0CRAWeRi9TsY9QLDy7eTnrOcRx2Ichuo8jlfi2voIis3ALqhgfjcrkoMobM3ALsIjhd/v2Fz2aDOiEO8gqLiAxxEB0WhE0EAWw2ochlyC8somFUKHabYLMJQTYbESF297IINhvkHncSGxGMzWZj5qwvWbF2Eyu37t9T2bisTBQpQNMSy02A1EpvLXMnbJsD+1bBzwv//L7NARFxEBwBIZFQp9Hvj4j6EO55hERBUJg7GQSFgSPE/dzmKOWkp5R/KixycfhYIYVFLgqdhoN5BRS5XGTnFXKssAhBcLrcJ3ZnkWFPVh7RYUGk5eSTnVdIaJCNtXsO0TQmnAKni9XJB2kZG0GRy1BY5OLXjKOEOGwE220UulzkF7qs/pX/LL/0l4PsQmGRoXn9cPZk5XHBWfWpFx6MzSbYBOwiiAh2G9hE2Hcoj/aNowm227DbhOy8Qs6OiyTILghCVJiDJvXCEXGv736AnPTT5jl/RIY4sNnEc2J3v158kvesfzp2797NlVdeSfPmzfn66685/+r7gNPbrpWJYi4wQkRmAN2Bw8aYP3U7lSt7Hyx7CdZ/5OkiAuwh0OpyOKMd1G0OcedCXDsICq3K+JU6bYc9J+7c44UcyXeScugYdptQWOTiuNPF3qw8guw2fs3IJSYimONOF+v3HqJ5/XCKXAany32iX7X7IGc2iODg0QIO5RVQWFQ135y3px0pfr4lNecP7x13umM8WWSIw30CtAl2keITcH6hixb1w3F4TrgOm3DwaAFhwXbio8PcDXHPZ06cqH/LOU7TmDDqhAYR5D6LcvS4kxb1w7HbbDhswtECJ42iQwl22IpfcxlDdFgQoUF2HDZ3iyM2MoTQINtpn4j92WOPPca4ceNwuVzEx8dX2XZ9lihEZDpwCRArIinAP4EgAGPMZGA+cBWwE8gDbq/wTjJ+hncvg/zD7i6f826GZufD2b0hukkV/SZKla/IZcjOKyAzt4D0I/k4XYbjhS7W7jnI/uxjRIY42JKaQ2SIg9XJB2kUHcb+7GOV3l/JE3hpr9nE3YXTMCqUIIeQc8xJsMNGy9gI9h86xjlnRBIR4sDh6b6wi5CWk0/HJtEcL3RRNzyIBnVCKCwyNK4XRrDdhgjUCw/GYRMcdhtBdnGfwO2Cw+ZersknYX+2efNm+vbtS0pKCmFhYUyZMoWhQ4dW2fZ9liiMMTeV874B7q/0DnLT4eMb3Emi5cXQ71WIPbvSm1O1W4HTRUbucZxFLgqL3F00v+UcJ6/AyW85+fy0+yD1woP5aXcWzWLCOXq8iK0HcmgYFUpazin6OMpQMkkE2YUzokJJOXSMTs3qkp1XSLv4KILsNlzGUOB00bx+BAZD03rhBNttHCssIr5uGA5PF4bDJgQ7bNSPDKFRdCghjpr9zVn90bp160hJSeHyyy9n9uzZhIeHV+n2rex6Oj1LX3APVDfsCDdNd489qFrLGMOR4052ZRzlWEERBw67u3CcRcbdr+5ysTM9l8N5hQQ7bCRnHWVvVh4GOHC4Yif633KOFz8vmSRCg2w0jAolKiyIvQfz6NYiprgf/8wGETSpF05hkfukHxnioF5EEGfUCcVm0xO6qrgffviB119/nc8++4xbb72VHj160KpVK5/sKzATRWE+bJntfn71W5okahhjDIfyCjl8rJCMI8dxulzsP3QMY9zdK3sP5mG3wYZ9hyv1bb4sTWPCCLK7+7nTDufTMDqU7i3r06ReGE6X4awGkdgEGtQJIchuIzosiIgQB1GhDhx2rYijfM/pdHLzzTfz2WefAZCUlERCQoLPkgQEaqL4cRIcOwgNO7gHrFVAMMaQnJXHwaPHWZ18iF0ZuaQfOU5eQRG/pucSHR5EfkERqRX8hn+ymIhg2jeOZndmLl2bx3j61KX4ipWGUaG0OiMSmwgNo0NpWi+cmMhgokKDqug3Vco3FixYwI033khOTg7169dnzpw5JCQk+Hy/gZkots5x/+z1pF6y6mdy8gs5kJ3Pil8z+Xrrb0SHBbFyVxZ1Qh3sO1j24G3W0YI/vRYTEUx0WBBNY8JJz8nngrNiiYkIIjo8mAaRwUSEOGgUHUZ83VDCgwPzz1kpb2RmZtKvXz8Ahg8fzttvv43NVj2t2MD7n2UM/LYFEGje0+poap3DeYWs3JXJnqw81u09xN6DxzDGlHoVTknZeYXFzx02oXOzemQfK+CiVg1oFB1Ks5hw4uuGERUaRHR4ENFh+u1eKYBFixZx+eWXExsby7PPPsvAgQPp2LFjtcYQeImi4Ki7/MYZ7SE0yupoaqSUQ3nsSDvCltQcDuUVsDvzKBtTDnOwlG/8pQl22MBAs/rhXHBWfXq1jiMqzEHrhlFEhgTen5xSVsjOzqZfv36sWLGCe+65h7feeounnnrKklgC739t4VH3z+YXWBtHDbB5/2E2pGRT6HSxMyOXOUmpHMl3lvu5to2iaBsfxRlRIYQ67DSNCadhdChnxkbQoE6IXpap1Gl66623GDVqFAUFBbRq1YpHH33U0ngCMFF4BjrPaG9tHAEkr8DJppTDfPTTXtYmH/R6sPiyNnGEOOz0bhNHk3rhRIY4ODsu0t1iUEr5RO/evVmyZAl2u51///vfPP7441aHFICJwpkP2CCujdWR+B1jDDnHnKzfd4jvfs5gw75s1u3NLvMzdUIdtI+Ppm18FG0aRdG9ZQzxdcOw67X9SlUrl8uFzWbjggsuICsri4ULF9KwYUOrwwJA3DdIB46uTUPNmjtD4O9bIbqx1eFYylnkYnZSKptSspm7IZVDJQaMS9OhcTStG9bhsjZxdG5WT7uJlPIDv/76K1dccQUtWrRg8eLFPtuPiKw1xnStzGcDr0VR5ARCIDLO6kiqXXZeAfM3pbE6+SAb9mWzK/PoKde9qFUsUaFB9DirPpeeG0d83bBqjFQp5Y2HHnqI1157DWMMzZs3tzqcUwq8RIGB4Ej3fA41XE5+IRv3HeaTNfv4csOpK7DXCXVwy/nN6dA4msvbnkGQ3iGslF9LSkqiX79+pKamEh4eztSpUxk8eLDVYZ1SACYKIDTa6gh8Iiv3ON/uyGDexlRW7MyioOjPJZzrhDpoUT+Cazs3pufZsbSKi9TuI6UCzObNm0lNTaVv377MmjWL0FD/ngIhMBNFWD2rI6gyK3ZmMjtpP2uSD5XaldQsJpxG0aG0i4/mgd5nUzc82IIolVKna/ny5bz++uvMnDmTW265hZ49e9KyZUurw/KKJopqZoxh3d5DfLUxjcXbf2NPVt6f1rnw7Fhu/EtTLj03TstSKBXgCgoKuOmmm5g1axYAGzdupGPHjgGTJCBQE4XDv5tppUnPyefDlXuYsXovmbm/3+FsE+jWIoZ28dHc3+ss6keGWBilUqoqzZs3j5tvvpkjR44QGxvL3Llzq738RlUIzEQRQAPZm/cf5vrJK/40p3D9iGBGXno2A86L1+SgVA2UmZnJwIEDAbj77ruZNGlStRXxq2qBmShs/h/2zvRcnvxiEz/tPlj8WqPoUHq3ieOBS1sRFxV4rSKlVPnmz59Pnz59iI2N5fnnn2fAgAG0bx/YlST8/4xbGj9tURQWufjoxz3MSUolaV928eu9Wjfgqf5tOatBpHXBKaV86uDBg/Tt25dVq1YVF/Hzh/IbVSFAE4V/Xfnzyeq9PPfVtj8V1GteP5z7LjmLId2aWRSZUqo6vPHGGzz00EMUFhbSunVrxowZY3VIVSowE4XDP/r0nUUu7vrvWpZsT//D67f2aM7fLzuHehH+ldCUUlWvV69eLF26FIfDwUsvvcQjjzxidUhVLjATRZD1c2SvTj7I/dPWkX7kOOC+eunjv53P+WfWtzgypVR1OFHE7+KLLyYnJ4cFCxYQF1czSwsFXlHAeLtZ89+x0PtpS/afciiPNxbv5JM1+wD3bG0v39CRazo1sSQepVT12rFjB3379qVFixYsWbLE6nC8VruKAoJlXU+v/G8HbyzZWbxsE1j15GXEaBeTUjWey+XioYceYvz48RhjOPPMM60OqdoEZqKw2at1d8edRTw+axOz1u0HICYimJeu68hlbc+o1jiUUtZYt24d/fr1Iy0tjfDwcD744AOuv/56q8OqNoGZKKrxqqd3l+/mX/O2Fi/fdkEL/tG/LTad2EepWmPbtm2kpaXRr18/Zs6c6fdF/KpaYN4maPP9fRTGGJ6evfkPSWLKrV0ZO7CdJgmlaoGlS5dyzTXXADB06FCSk5OZN29erUsSEKgtCh93Pe3JOsplr35HYZF7oP+v5zfnnwPa4tB5HpSq8QoKChg8eDBz5swBfi/i588TC/laYJ75fDiY/dGPe7j45aXFSeKxPufyr6vba5JQqhaYM2cOMTExzJkzh7i4OFatWhWQRfyqWmC2KHw0RpG47Ff+PX978fLU27rR69yaeV20UuqPMjMzi7uaRowYwfjx4wO2iF9VC8yj4INaT0u2/1acJIb1aM7uF67SJKFULTB37lxcLhexsbG8+OKLbNmyhTfeeEOTRAkBeiSqbjDZGEPisl+54/01APTv2IixA9vp9KJK1XCZmZl069aNQYMGcd999wHw6KOP0qZNG4sj8z+BmSiq8CT+13dXFbckLj03jteGJGiSUKqGe/3114mPj2fNmjW0adOGJ554wuqQ/FpgjlFUwXwUuced3DdtHct3ZgJwbsM6vDusqyYJpWq4iy++mGXLluFwOHjllVcYPXq01SH5vVqbKEZ/ksSynzMAeLzvudx98VmnvU2llP86UcSvV69eHD16lIULFxIbG2t1WAHBp11PItJHRHaIyE4R+VOBdhGJFpEvRWSDiGwRkdu93PJpxbVwcxr/2/oboElCqZpu27ZttGjRgt69ewMwduxY1qxZo0miAnyWKETEDrwJ9AXaAjeJSNuTVrsf2GqMOQ+4BHhFRMq/9vU0uofSc/IZOX0dANd0aqxJQqkayuVycf/999OuXTv27NlDcLAW76wsX3Y9/QXYaYzZBSAiM4BBwNYS6xigjrgHBiKBg4Dz5A39WeUShTGGAROXU1hkaBYTzr+uDux5bJVSpVu9ejX9+/cnPT2diIgIpk2bxqBBg6wOK2D5suupMbCvxHKK57WSJgJtgFRgE/CgMcZ18oZE5C4RWSMia04noKdmb+a3HPdEQ5Nv6UJkSGAO0SilyrZz507S09MZNGgQBw8e1CRxmnyZKEr72n/yLElXAklAPJAATBSRqD99yJhEY0zX4kk3pOJhp+fk89naFAD+OaAtbeP/tBulVABbvHgxV199NQA33XQT+/btY/bs2drlVAV8mShSgKYllpvgbjmUdDswy7jtBHYD55a75UrcMfn64l8ocLq4pHUDbu/ZssKfV0r5p/z8fPr3789ll13GnDlz2Lx5MwBNmuisk1XFl4liNdBKRFp6BqhvBOaetM5eoDeAiJwBtAZ2lb/pio1RrNiZycc/7QVg5KVnV+izSin/9fnnnxMbG8tXX31Fw4YNWb16Ne3b69hjVfNZojDGOIERwCJgG/CpMWaLiNwjIvd4VvsXcIGIbAIWA48ZYzLL3XgFrnoyxvDo5xsB6NK8Hl2ax1To91BK+afMzExuuOEG8vLyeOCBB9i/fz9du1ZqSmhVDp+O5hpj5gPzT3ptconnqcAVvozhox/3kHLoGACThnb25a6UUtXgiy++YNCgQcTGxvKf//yHgQMH0rp1a6vDqtECs9aTl4wxTPN0OfXr2IgzomrfzFRK1RTp6el06dKFa6+9triI3yOPPKJJohrU6EQxb+MBtqcdoW54EK8OPs/qcJRSlfTyyy/TuHFj1q1bR/v27XnqqaesDqlWqdE3EiQuc4+L/+2iMwlx+Hb6VKWUb1x00UUsX76coKAgJkyYwMiRI60OqdapsYli9vr9bNp/mNAgG7f3bGF1OEqpCnI6nTgcDi6//HIKCgpYsGABMTF6MYoVamzX09eeon/t46MJD66x+VCpGmfLli00a9asuIjfP/7xD3766SdNEhaqkYmiyGX4atMBAB7o3criaJRS3nC5XNx777106NCBffv2ER4ebnVIyqNGftWetc5dqiPILvQ8W0sJK+XvfvrpJwYMGEBGRgaRkZFMmzaNgQMHWh2W8qiRieK9H5IBGNq9OXabzlinlL9LTk4mIyODa6+9lunTp2t9Jj8ToF1Ppz75p2YfY9uBHADu0bkmlPJbixYton///gAMGTKE/fv38/nnn2uS8EOB2aIoo4THgs1pxc8bRusNdkr5m/z8fK655hoWLlyIiLB582bat29PfHy81aGpUwjQFsWpTVzyCwDPDmpncSRKqZN9+umnxMTEsHDhQho1alR8A53ybzUqUezOPMqhvEIAruusJYaV8ieZmZnceOONHD9+nIcffpjU1FQSEhKsDkt5IUATReldT4u2uLudureMIUJnr1PKL8ycOROXy0VsbCzjxo3j559/5uWXX7Y6LFUBgZkoTjFGMX2VuwDg4K5NS31fKVV90tLSSEhI4IYbbuCee9wzC4wePZqzztKLTAJNYCaKUloULpch3TMfdo+z6ld3QEqpEl588UWaNm3Khg0b6NChA2PHjrU6JHUaakz/zKb9hzlWWIQIxNcNszocpWqtnj17smLFCoKDg3nzzTeLS4KrwOV1ohCRCGPMUV8GczrW7DkEQP+OeomdUlY4UcTvyiuvxBjD/PnzqVu3rtVhqSpQbteTiFwgIltxT2eKiJwnIpN8HlkFff9LBgA9tdtJqWq1ceNGmjRpwqWXXgq4i/itWLFCk0QN4s0YxWvAlUAWgDFmA/B/vgyqMpL2ZQNw/pmaKJSqDi6Xi+HDh5OQkMD+/fuJioqyOiTlI151PRlj9skfrzQq8k04lXPg8DGy8wqx24Tm9bXipFK+9sMPPzBo0CCysrKIiopixowZ9O3b1+qwlI94kyj2icgFgBGRYOABPN1Q/uK7He5up5axEUgZ5T2UUlVj//79HDx4kBtuuIGPP/4Yh6PGXBejSuFN19M9wP1AYyAFSAD86jKGDSnZALSoH2FtIErVYAsWLKBfv34ADB48mNTUVD799FNNErWAN//CrY0xQ0u+ICI9gR98E1LFGGOKZ7O7v5feyKNUVcvLy+Oaa67hf//7HyLCli1baNeuHQ0bNrQ6NFVNvGlRvOHla9WnRPdSyqFjZOYWEBFsJ6FpXetiUqoGmj59OrGxsfzvf/+jcePGbNy4kXbttOBmbXPKFoWI9AAuABqIyOgSb0UBdl8H5q0daUcAOKdhHR2fUKoKZWZmMnToUESERx55hJdeesnqkJRFympRBAORuJNJnRKPHOB634fmnR2/uRNFqMNvcpdSAW369OnFRfxeffVVdu7cqUmiljtli8IY8x3wnYi8b4zZU40xeeH3lsOuDPfN4l2a17MqGKVqhNTUVPr06cOmTZv49ttvSUxMZNSoUVaHpfyAN2MUeSLysojMF5ElJx4+j8xLq5KzADhDZ7NTqtKee+45mjVrxqZNm0hISODZZ5+1OiTlR7y56mka8AnQH/elssOADF8GVa4SYxHHCtz3/l2gpTuUqpQePXrw448/EhwczKRJk7jrrrusDkn5GW8SRX1jzLsi8mCJ7qjvfB1Y2dyJ4recfDJzCwiyC2fG6j0USlVEQUEBwcHB9O/fH4fDwVdffaVlOFSpvOl6KvT8PCAi/USkE+AX84yeGJ/o0Dhar3hSyktJSUnEx8fTu3dvAJ588km+//57TRLqlLxJFM+JSDTwEPAw8A4wypdBlcuTFPZnHwOgcT2t76RUeVwuF3fccQedO3fmwIEDxMTEWB2SChDldj0ZY+Z5nh4GekHxndkW8iSKQ+5EEa8D2UqVafny5QwaNIiDBw8SFRXFp59+ypVXXml1WCpAnLJFISJ2EblJRB4Wkfae1/qLyApgYrVFWIafPfdQNKmnM9opVZbU1FQOHTrEjTfeSFZWliYJVSFldT29CwwH6gMTRGQqMA54yRjTyZuNi0gfEdkhIjtFZMwp1rlERJJEZEtFB8m/3uau8dSgjrYolDrZ3Llz6dOnD+Au4peWlsb06dO1iJ+qsLL+YroCHY0xLhEJBTKBs40xad5sWETswJvA5birzq4WkbnGmK0l1qkLTAL6GGP2ikhcRYKvFx7EbznHaaxzZCtVLDc3l6uvvprFixf/oYhfXFyF/nspVaysFkWBMcYFYIzJB372Nkl4/AXYaYzZZYwpAGYAg05a52ZgljFmr2c/6V5tWQSXy/BbznEAWp0RWYGwlKq5PvzwQxo0aMDixYuLb6DTIn7qdJWVKM4VkY2ex6YSy5tEZKMX224M7CuxnOJ5raRzgHoislRE1orIraVtSETuEpE1IrLG8wrpR9xJIjTIRmiQ1nlSKj09ndtuu42CggKeeOIJ9uzZo0lCVYmyup7anOa2S7uxwZSy/y5AbyAMWCkiPxpjfv7Dh4xJBBIBusbbDcD2tBwA6keEnGaYSgW2adOmMWTIEOLi4hg/fjwDBw6kefPmVoelapCyigKebiHAFKBpieUmQGop62QaY44CR0VkGXAe8DNlEUjOdN9s10NLd6haKiUlhT59+rBlyxa+++47EhMTGTlypNVhqRrImxvuKms10EpEWnrm2r4RmHvSOnOAi0TEISLhQHe8nI978Xb3cEaYdjupWuiZZ56hRYsWbNmyhS5duvDcc89ZHZKqwXx2nZwxxikiI4BFuCc6es8Ys0VE7vG8P9kYs01EFgIbARfwjjFmszfbP3HvRH5hkU/iV8pfde/enVWrVhESEsLbb7/NnXfeaXVIqobzKlGISBjQzBizoyIbN8bMB+af9Nrkk5ZfBl6uyHYBDh11l6C6uHWDin5UqYDjcrlwOp0EBwdz9dVXExoaypdffqn1mVS1KLfrSUQGAEnAQs9ygoic3IVU7Q4cdpfviNOb7VQNt2bNGho3bsyll14KwOOPP853332nSUJVG2/GKMbiviciG8AYkwS08FVA3ko9nA9AI63zpGool8vFrbfeSrdu3UhLS6NBA209K2t40/XkNMYc9qcy3sZAdl4BAA3q6OWxquZZtmwZgwYNIjs7m7p16zJz5szisuBKVTdvWhSbReRmwC4irUTkDWCFj+MqU35hEYVFhrAgu95sp2qkjIwMDh8+zNChQ8nIyNAkoSzlTaIYCbQDjgMf4y43PsqHMZUrJ98JgNPlsjIMparUF198wRVXXAHAddddR3p6Oh999JEW8VOW8+YvsLUx5kngSV8H461jhe5EEerQ1oQKfLm5uQwYMIClS5f+oYhfbGys1aEpBXjXonhVRLaLyL9ExC8Kx2TnuRNF/chgiyNR6vR88MEHxMbGsnTpUpo3b16cJJTyJ+UmCmNML+ASIANI9BQFfMrXgZXl6PETiUIHslXgSk9P5/bbb6ewsJCnnnqK5ORk2rQ53RJrSlU9r0p4GGPSjDETgHtw31PxD18GVZ7fctyXxnZoHG1lGEpVyocffojT6SQuLo6JEyeye/du/vWvf1kdllKn5M0Nd21EZKyIbMY9BeoK3AX+LJOT774rWy+NVYFk7969tGnThmHDhnHfffcBcN9999GsWTOLI1OqbN60KKYCh4ArjDEXG2Pe8nqCIR/ZmZ4LQGSIXg2iAsPTTz9Ny5Yt2b59O127duXf//631SEp5bVyz7TGmPOrI5CKsHlu/itw6uWxyv9169aNNWvWEBoayuTJkxk2bJjVISlVIadMFCLyqTFmsGd2u5ITDglgjDEdfR7dKWQddd+V3ax+uFUhKFUml8tFQUEBoaGhXHfddURFRTFnzhwiI3XaXhV4ympRPOj52b86AqmIUIe7xywmQi+PVf5n9erV9O/fn7PPPpsffviBMWPGMGbMGKvDUqrSTjlGYYw54Hl6nzFmT8kHcF/1hFe6Y545KKLDgqwMQ6k/cDqdDB06lL/85S+kp6fTqFEjq0NSqkp4M5h9eSmv9a3qQCoi97g7UUSFaqJQ/mHp0qXExsby8ccfU69ePb799ltmzpxpdVhKVYmyxijuxd1yOFNENpZ4qw7wg68DK0ue54a7qDC96kn5h6ysLHJychg2bBjvvfceNpsvZxlWqnqVdab9GFgAvACU7GA9Yow56NOoylFQ5CLYbtP5spWlPv/8c9566y2++eYbrrvuOjIzM4mJibE6LKWqXFmJwhhjkkXk/pPfEJEYq5NFVJgDf5ojQ9UeOTk59O/fn++///4PRfw0Saiaqqz28ceen2uBNZ6fa0ssW0pvtlNWeOedd4iLi+P777/nzDPPZMeOHVrET9V4pzzbGmP6e362rL5wvKcTFqnqlp6ezl133YXNZuOZZ57hH/+wtOSZUtXGm1pPPUUkwvP8FhF5VUQsL04ToolCVZN33323uIjfW2+9xd69ezVJqFrFm0sz3gLyROQ84FFgD/Bfn0blhb1ZR60OQdVwe/bsoXXr1gwfPry4iN/dd99NfHy8xZEpVb28SRROY4wBBgHjjTHjcV8ia6luLXTgUPnO448/zplnnsnPP/9M9+7defHFF60OSSnLeDMifEREHgf+ClwkInbA8jvdIkN1MFv5RpcuXVi3bh2hoaFMmTKFW265xeqQlLKUN2fbIcDNwB3GmDTP+MTLvg2rfBHBmihU1SlZxG/IkCHExMQwZ84cwsO18KRS3kyFmgZMA6JFpD+Qb4z50OeRlSPYoXe+qqqxcuVKGjZsSO/evQF49NFH+frrrzVJKOXhzVVPg4FVwA3AYOAnEbne14GVJ0QThTpNTqeTG2+8kQsuuICMjAwaN25sdUhK+SVv+m+eBLqdmNVORBoA3wCWVjw7lFdg5e5VgFu8eDHXXnstOTk5xd1MF154odVhKeWXvPlabjtp6tMsLz/nMwahTaMoK0NQAS4nJ4fc3FzuvPNOMjIyNEkoVQZvWhQLRWQRMN2zPASY77uQvHOsoMjqEFSA+eSTT5g8eTLffvst11xzDVlZWdStW9fqsJTye97Mmf2IiFwLXIh7GtREY8wXPo+srJiApjE60Ki8k52dTb9+/VixYgU2m624iJ8mCaW8U9Z8FK2AccBZwCbgYWPM/uoKrDw6mK288fbbb/PAAw9QUFDAWWedxaJFizjrrLOsDkupgFLW2fY9YB5wHe6KsW9US0Re0grjqjzp6ence++9FBUV8dxzz7Fz505NEkpVQlmJoo4xZooxZocxZhzQoppi8kpYkN5wp0qXmJhYXMRv8uTJ7N27lyeffNLqsJQKWGUlilAR6SQinUWkMxB20nK5RKSPiOwQkZ0iMqaM9bqJSFFF7s+ICNHqseqPdu/ezTnnnMPdd9/NvffeC8Bdd92lRfyUOk1lfS0/ALxaYjmtxLIBLi1rw56aUG8ClwMpwGoRmWuM2VrKev8BFlUkcL0zW5X06KOP8sorr+ByuejRowcvv2x5lRmlaoyyJi7qdZrb/guw0xizC0BEZuCuQLv1pPVGAp8D3SqycYdNBymUW+fOnVm/fj1hYWG8++673HTTTVaHpFSN4suO/sbAvhLLKUD3kiuISGPgGtytk1MmChG5C7gLoEsjd0siQqdCrdVKFvG76aabaNCgAV988YXWZ1LKB3zZf1PaV35z0vLrwGPGmDLvnjPGJBpjuhpjup54LdiuXU+11fLly4mLi6NXL3ej95FHHmHRokWaJJTyEV+ebVOApiWWmwCpJ63TFZghIsnA9cAkEbnam43rGEXt43Q6uf7667nooovIysqiZUu/nM5dqRqn3P4bERFgKHCmMeZZz3wUDY0xq8r56GqglYi0BPYDN+Ke16KYMab4f7qIvA/MM8bM9ibwEIde9VSbfP3111x33XUcOXKE2NhYZs+eTc+ePa0OS6lawZuv5ZOAHsCJEcIjuK9mKpMxxgmMwH010zbgU2PMFhG5R0TuqWS8xbRFUbvk5eVx9OhR7r77bn777TdNEkpVI29GhLsbYzqLyHoAY8whEQn2ZuPGmPmcVEDQGDP5FOve5s02AUKDtDVRG0yfPp23336bpUuXMmjQIA4dOkRUlFYNVqq6eZMoCj33Ohgono/C5dOoyhFk10tja7Ls7Gz69u3Ljz/+iM1mY9u2bbRp00aThFIW8ab/ZgLwBRAnIs8Dy4F/+zSqcmj5jppr0qRJxMXF8eOPP9KqVSt27txJmzZtrA5LqVrNmzLj00RkLdAb9yWvVxtjtvk8sjJoi6JmSktLY8SIEdjtdl544QXGjDll1RelVDXyZs7sZkAe8CUwFzjqec0ydr0ru0aZNGkSTqeThg0bkpiYyL59+zRJKOVHvOnD+Qr3+IQAoUBLYAfQzodxlclh0yueaoJffvmFPn36sGvXLtavX8+UKVMYPny41WEppU5S7hnXGNPBGNPR87MV7hpOy30f2qnlHndauXt1mlwuF6NHj6Z169bs2rWLiy66iFdeecXqsJRSp1DhUWFjzDoRqVABv6qmYxSBrUuXLiQlJREeHs7UqVMZPHiw1SEppcrgzZ3Zo0ss2oDOQIbPIvJCXFSIlbtXleByucjPzyc8PJxbbrmFRo0aMWvWLEJDQ60OTSlVDm86++uUeITgHrMY5MugymPXeVADyrJly2jQoAGXXuqewuShhx5i/vz5miSUChBltig8N9pFGmMeqaZ4vOLQyrEBoaCggBtvvJEvvvgCgLPPPtviiJRSlXHKRCEiDmOM09tpT6tTyqE8q0NQ5Vi0aBHXX389ubm5NGjQgC+//JLu3buX/0GllN8pq0WxCvd4RJKIzAU+A46eeNMYM8vHsZ3SWQ0irdq18tLx48fJy8vj3nvvZeLEidj0kmalApY3Vz3FAFm4Z6E7cT+FASxLFDYdo/BLH374IVOmTOH7779n4MCBWsRPqRqirEQR57niaTO/J4gTTp6prlrZ9M5sv3Lw4EH69OnD6tWrtYifUjVQWf0BdiDS86hT4vmJh2X0qif/MX78eBo2bMjq1as599xz2bVrlxbxU6qGKatFccAY82y1RVIBmif8Q1paGn//+9+x2+2MGzeOhx56yOqQlFI+UFaLwm9Pxw7terLUxIkTi4v4vfvuu+zfv1+ThFI1WFktit7VFkUF6RiFNXbs2EGfPn1ITk5mw4YNTJkyhdtvv93qsJRSPnbKFoUx5mB1BlIRLmPpWHqt43K5ePDBB2nTpg3JyclcfPHFWsRPqVokIKeKO3S00OoQapVOnTqxceNGIiIi+OCDD7juuuusDkkpVY0CMlE0qqs1gnzN5XKRl5dHZGQkw4YN49tvv+Xzzz8nODjY6tCUUtUsIG+XDcigA8iSJUuoX79+cRG/0aNH8+WXX2qSUKqWCshzrg5m+0ZBQQGDBg2id+/eZGdn07ZtW6tDUkr5gYDsehK9kaLKzZ8/n8GDB3P06FHOOOMM5s2bR9euXa0OSynlBwKyRaF3Zlc9p9NJfn4+I0eOJDU1VZOEUqpYQCYKTRNVY+rUqfTs2ROAgQMHkp2dzYQJE7TSq1LqDwKy60nHKE5PZmYmffr0Ye3atX8o4hcZqeXblVJ/FpBfHbXnqfJeffVVGjVqxNq1a2nTpg27d+/WIn5KqTIFZKKwa6KolLS0NB5++GEAXnvtNbZu3UqzZs0sjkop5e8CMlHoVU8V8/rrrxcX8Zs6dSoHDhxg1KhRVoellAoQATlGYdcxCq9s27aNPn36sHfvXrZu3UpiYiLDhg2zOiylVIAJyBaFXh5bNpfLxf3330+7du3Yu3cvvXv35tVXX7U6LKVUgArIFoWOZpctISGBTZs2ERkZybRp0xg4cKDVISmlAlhAJgrtefqzEzfMRUZGcscdd/Ddd9/xySefaH0mpdRp82nXk4j0EZEdIrJTRMaU8v5QEdnoeawQkfO8227VxxrIvv76a2JjY+nVqxcAo0aN4osvvtAkoZSqEj5rUYiIHXgTuBxIAVaLyFxjzNYSq+0GLjbGHBKRvkAi0L28bWuLwi0/P5/rr7+er776CoD27dtbHJFSqibyZdfTX4CdxphdACIyAxgEFCcKY8yKEuv/CDTxZsOiRTyYN28eQ4YMIS8vj4YNG/LVV1/RuXNnq8NSStVAvux6agzsK7Gc4nntVO4EFpT2hojcJSJrRGSNZ7nKggxkx48fZ9SoUezfv1+ThFLKZ3zZoijtbF7qZNci0gt3oriwtPeNMYm4u6XoGm83tTVNvPPOO7z77rusXLmS/v37k5OTQ3h4uNVhKaVqOF8mihSgaYnlJkDqySuJSEfgHaCvMSbLmw3XtuKm6enp9OnTh/Xr12O324uL+GmSUEpVB1+eclcDrUSkpYgEAzcCc0uuICLNgFnAX40xP3u74drU9fTSSy/RuHFj1q9fT4cOHdi7d68W8VNKVSufJQpjjBMYASwCtgGfGmO2iMg9InKPZ7V/APWBSSKSdGIMojy1ZTA7LS2NMWPGICK88cYbbNy4kfj4eKvDUkrVMj694c4YMx+Yf9Jrk0s8Hw4Mr+h2a/rlsa+88gojR46kYcOGfPjhh1x11VXExMRYHZZSqpYKyDuza2rP0+bNm+nbty8pKSls376dKVOmcMstt1gdlqqlCgsLSUlJIT8/3+pQVAWEhobSpEkTgoKCqmybgZkorA6girlcLu677z4SExMxxnD55Zczfvx4q8NStVxKSgp16tShRYsWtWpcMJAZY8jKyiIlJYWWLVtW2XYDMlHkF7qsDqFKdezYkS1bthAZGcn06dPp37+/1SEpRX5+viaJACMi1K9fn4yMjCrdbkAmitCgwL8+1ul0kpeXR1RUFH/729/4/vvv+fjjj7U+k/IrmiQCjy/+zQLyjOsI8BspFixYQP369bn00ksBePDBB5k5c6YmCaWUXwrsM26AycvLo0+fPlx11VUcOXKETp06WR2SUn7v+eefp127dnTs2JGEhAR++uknnE4nTzzxBK1atSIhIYGEhASef/754s/Y7XYSEhJo164d5513Hq+++iouV83qsq5OAdn1FIit4Tlz5nDTTTdx7Ngx4uPjWbBgAR07drQ6LKX82sqVK5k3bx7r1q0jJCSEzMxMCgoKeOqpp0hLS2PTpk2EhoZy5MgRXnnlleLPhYWFkZSUBLgrG9x8880cPnyYZ555xqLfJLBpoqgmQUFBFBQU8PDDD/Pyyy9bHY5SFdJizFc+2W7yi/3KfP/AgQPExsYSEhICQGxsLHl5eUyZMoXk5GRCQ0MBqFOnDmPHji11G3FxcSQmJtKtWzfGjh2r4y6VEKBdT4HxD/3222/Tvbt7eo2rrrqK3NxcTRJKVcAVV1zBvn37OOecc7jvvvv47rvv2LlzJ82aNaNOnTpeb+fMM8/E5XKRnp7uw2hrroBsUfh7nkhLS+PKK69k48aN2O12duzYQevWrYu//SgVaMr75u8rkZGRrF27lu+//55vv/2WIUOG8MQTT/xhnalTpzJ+/HiysrJYsWIFTZs2LXVbxpRavFp5IUBbFP7rhRdeoEmTJmzcuJGOHTuSkpJC69atrQ5LqYBlt9u55JJLeOaZZ5g4cSJffvkle/fu5ciRIwDcfvvtJCUlER0dTVFRUanb2LVrF3a7nbi4uOoMvcYIyEThrw2KtLQ0nnzySex2O5MmTWLDhg00bNjQ6rCUClg7duzgl19+KV5OSkqidevW3HnnnYwYMaK4vEhRUREFBQWlbiMjI4N77rmHESNG6PhEJQVk15O//WO/+OKLjB49moYNGzJt2jT69u1L3bp1rQ5LqYCXm5vLyJEjyc7OxuFwcPbZZ5OYmEh0dDRPP/007du3p06dOoSFhTFs2LDi6srHjh0jISGBwsJCHA4Hf/3rXxk9erTFv03gkkDrt+sabzeJs76h8/m9rA6FpKQk+vXrR2pqKn/7299ITEy0OiSlqsyJCbJU4Cnt305E1hpjulZme9r1VAkul4vhw4fTuXNnUlNT6dOnDxMmTLA4KqWU8o2A7HqyWvv27dm2bRtRUVHMmDGDvn37Wh2SUkr5jCYKL5Us4nfvvffyww8/8NFHH+Fw6CFUStVsgdn1VM19T/PmzSMmJqa4iN/IkSOZMWOGJgmlVK0QkGe66soTeXl5XH311Xz99deICN26daumPSullP8IyERRHUoW8WvSpAkLFiygffv2VoellFLVLiC7nqqjSREcHExhYSGPPvoo+/bt0yShlAVOlAtv3749AwYMIDs7u0q2+/777zNixIgq2VZJl1xyCa1bty4ufT5z5swq3wdAcnIyH3/8sU+2XZqATBTio0wxceJEunZ1X2bct29fjhw5wn/+8x+f7EspVb4T5cI3b95MTEwMb775ptUhlWvatGkkJSWRlJTE9ddf79VnnE5nhfZR3YkiILueqnowOzU1lSuvvJLNmzfjcDi0iJ9SJxsb7aPtHvZ61R49erBx40YAVq1axahRozh27BhhYWFMnTqV1q1b8/777zN37lzy8vL49ddfueaaa3jppZcAd/HAF154gUaNGnHOOecUly7fs2cPd9xxBxkZGTRo0ICpU6fSrFkzbrvtNsLCwti+fTt79uxh6tSpfPDBB6xcuZLu3bvz/vvvexX3wYMHueOOO9i1axfh4eEkJibSsWNHxo4dS2pqKsnJycTGxjJ+/Hjuuece9u7dC8Drr79Oz549+e6773jwwQcBd1WKZcuWMWbMGLZt20ZCQgLDhg3j73//u9fHsTICMlFUpWeffZZnn32WoqIiOnXqxMKFC7VwmFJ+pqioiMWLF3PnnXcCcO6557Js2TIcDgfffPMNTzzxBJ9//jngrpiwfv16QkJCaN26NSNHjsThcPDPf/6TtWvXEh0dTa9evYpnmBwxYgS33norw4YN47333uOBBx5g9uzZABw6dIglS5Ywd+5cBgwYwA8//MA777xDt27dSEpKIiEh4U+xDh06lLCwMAAWL17M2LFj6dSpE7Nnz2bJkiXceuutxZMqrV27luXLlxMWFsbNN9/M3//+dy688EL27t3LlVdeybZt2xg3bhxvvvkmPXv2JDc3l9DQUF588UXGjRvHvHnzfHvgPWp1okhNTWXs2LEEBwczefJkhg8fbnVISvmnCnzzr0onajYlJyfTpUsXLr/8cgAOHz7MsGHD+OWXXxARCgsLiz/Tu3dvoqPdLaC2bduyZ88eMjMzueSSS2jQoAEAQ4YM4eeffwbcs+jNmjULgL/+9a88+uijxdsaMGAAIkKHDh0444wz6NChAwDt2rUjOTm51EQxbdq04i5sgOXLlxcnsUsvvZSsrCwOH3Yfz4EDBxYnlW+++YatW7cWfy4nJ4cjR47Qs2dPRo8ezdChQ7n22mtp0qTJaRzRygnMMYrT6HtyuVw899xzFBQUEB8fz4wZM0hPT9ckoZQfOjFGsWfPHgoKCorHKJ5++ml69erF5s2b+fLLL4uryALFXUrgHgw/0f/v7Xmj5HontmWz2f6wXZvN5vW4Qmn19E7sIyIiovg1l8vFypUri8c39u/fT506dRgzZgzvvPMOx44d4/zzz2f79u1e7bcqBWaiqOTnkpKSaNKkCU8//XTxFQ+DBw8mKiqq6oJTSlW56OhoJkyYwLhx4ygsLOTw4cM0btwYwKuxgu7du7N06VKysrIoLCzks88+K37vggsuYMaMGYC7NXDhhRdWaez/93//x7Rp0wBYunQpsbGxpZ5zrrjiCiZOnFi8fKJ76tdff6VDhw489thjdO3ale3bt1OnTp3i+TiqQ61IFC6Xi9tvv51OnTpx4MABrrrqKi3ip1SA6dSpE+eddx4zZszg0Ucf5fHHH6dnz56nnKyopEaNGjF27Fh69OjBZZddRufOnYvfmzBhAlOnTqVjx47897//Zfz48VUa99ixY1mzZg0dO3ZkzJgxfPDBB6WuN2HChOL12rZty+TJkwH3oHb79u0577zzCAsLo2/fvnTs2BGHw8F5553Ha6+9VqXxliYgy4z/98sltOlysdefadOmDdu3byc6OprPPvusuJ9TKXVqWmY8cFV1mfEaO5hdUFBAXl4edevW5f7772fFihV8+OGHWp9JKaUqKCC7nsrrfJozZw4xMTH07t0bcF/+9vHHH2uSUEqpSgjMM+cp8kRubi6DBg1iyZIliAgXXHBB9calVA1jjPG7qYdV2XwxnBCQiaK0P9vPP/+cW265hfz8fJo1a8bChQu1f1Wp0xAaGkpWVhb169fXZBEgjDFkZWVVeVWJwEwUpfzRhoeH43Q6eeKJJ3j++ectiEqpmqVJkyakpKSQkZFhdSiqAkJDQ6v8pryATBQnjB8/nv/+97+sWbOGvn37cvToUYKDg60OS6kaISgoiJYtW1odhvIDPh3MFpE+IrJDRHaKyJhS3hcRmeB5f6OIdC5tOyfLSE+nXbt2jBo1ig0bNrBjxw4ATRJKKeUDPksUImIH3gT6Am2Bm0Sk7Umr9QVaeR53AW+Vt93Dxw2XDRjM1q1b6dKlCwcOHKB169ZVHL1SSqkTfNmi+Auw0xizyxhTAMwABp20ziDgQ+P2I1BXRBqVtdHkbEOww84777zDmjVriI2N9U30SimlAN+OUTQG9pVYTgG6e7FOY+BAyZVE5C7cLQ6A40eOFW4ePny4FvKDWCDT6iD8hB6L3+mx+J0ei99VuuvFl4mitOvpTr7A15t1MMYkAokAIrKmsreh1zR6LH6nx+J3eix+p8fidyKyprKf9WXXUwrQtMRyEyC1EusopZSykC8TxWqglYi0FJFg4EZg7knrzAVu9Vz9dD5w2Bhz4OQNKaWUso7Pup6MMU4RGQEsAuzAe8aYLSJyj+f9ycB84CpgJ5AH3O7FphN9FHIg0mPxOz0Wv9Nj8Ts9Fr+r9LEIuDLjSimlqleAVo9VSilVXTRRKKWUKpPfJgpflf8IRF4ci6GeY7BRRFaIyHlWxFkdyjsWJdbrJiJFInJ9dcZXnbw5FiJyiYgkicgWEfmuumOsLl78H4kWkS9FZIPnWHgzHhpwROQ9EUkXkc2neL9y501jjN89cA9+/wqcCQQDG4C2J61zFbAA970Y5wM/WR23hcfiAqCe53nf2nwsSqy3BPfFEtdbHbeFfxd1ga1AM89ynNVxW3gsngD+43neADgIBFsduw+Oxf8BnYHNp3i/UudNf21R+KT8R4Aq91gYY1YYYw55Fn/EfT9KTeTN3wXASOBzIL06g6tm3hyLm4FZxpi9AMaYmno8vDkWBqgj7jkKInEnCmf1hul7xphluH+3U6nUedNfE8WpSntUdJ2aoKK/5524vzHUROUeCxFpDFwDTK7GuKzgzd/FOUA9EVkqImtF5NZqi656eXMsJgJtcN/Quwl40Bjjqp7w/Eqlzpv+Oh9FlZX/qAG8/j1FpBfuRHGhTyOyjjfH4nXgMWNMUQ2flc2bY+EAugC9gTBgpYj8aIz52dfBVTNvjsWVQBJwKXAW8LWIfG+MyfFxbP6mUudNf00UWv7jd179niLSEXgH6GuMyaqm2KqbN8eiKzDDkyRigatExGmMmV0tEVYfb/+PZBpjjgJHRWQZcB5Q0xKFN8fiduBF4+6o3ykiu4FzgVXVE6LfqNR501+7nrT8x+/KPRYi0gyYBfy1Bn5bLKncY2GMaWmMaWGMaQHMBO6rgUkCvPs/Mge4SEQcIhKOu3rztmqOszp4cyz24m5ZISJn4K6kuqtao/QPlTpv+mWLwviu/EfA8fJY/AOoD0zyfJN2mhpYMdPLY1EreHMsjDHbRGQhsBFwAe8YY0q9bDKQefl38S/gfRHZhLv75TFjTI0rPy4i04FLgFgRSQH+CQTB6Z03tYSHUkqpMvlr15NSSik/oYlCKaVUmTRRKKWUKpMmCqWUUmXSRKGUUqpMmiiUX/JUfk0q8WhRxrq5VbC/90Vkt2df60SkRyW28Y6ItPU8f+Kk91acboye7Zw4Lps91VDrlrN+gohcVRX7VrWXXh6r/JKI5BpjIqt63TK28T4wzxgzU0SuAMYZYzqexvZOO6bytisiHwA/G2OeL2P924CuxpgRVR2Lqj20RaECgohEishiz7f9TSLyp6qxItJIRJaV+MZ9kef1K0Rkpeezn4lIeSfwZcDZns+O9mxrs4iM8rwWISJfeeY22CwiQzyvLxWRriLyIhDmiWOa571cz89PSn7D97RkrhMRu4i8LCKrxT1PwN1eHJaVeAq6ichfxD0XyXrPz9aeu5SfBYZ4Yhniif09z37Wl3YclfoTq+un60MfpT2AItxF3JKAL3BXEYjyvBeL+87SEy3iXM/Ph4AnPc/tQB3PusuACM/rjwH/KGV/7+OZuwK4AfgJd0G9TUAE7tLUW4BOwHXAlBKfjfb8XIr723txTCXWORHjNcAHnufBuCt5hgF3AU95Xg8B1gAtS4kzt8Tv9xnQx7McBTg8zy8DPvc8vw2YWOLz/wZu8Tyvi7vuU4TV/9768O+HX5bwUAo4ZoxJOLEgIkHAv0Xk/3CXo2gMnAGklfjMauA9z7qzjTFJInIx0Bb4wVPeJBj3N/HSvCwiTwEZuKvw9ga+MO6ieojILOAiYCEwTkT+g7u76vsK/F4LgAkiEgL0AZYZY455urs6yu8z8kUDrYDdJ30+TESSgBbAWuDrEut/ICKtcFcDDTrF/q8ABorIw57lUKAZNbMGlKoimihUoBiKe2ayLsaYQhFJxn2SK2aMWeZJJP2A/4rIy8Ah4GtjzE1e7OMRY8zMEwsicllpKxljfhaRLrhr5rwgIv8zxjzrzS9hjMkXkaW4y14PAaaf2B0w0hizqJxNHDPGJIhINDAPuB+YgLuW0bfGmGs8A/9LT/F5Aa4zxuzwJl6lQMcoVOCIBtI9SaIX0PzkFUSkuWedKcC7uKeE/BHoKSInxhzCReQcL/e5DLja85kI3N1G34tIPJBnjPkIGOfZz8kKPS2b0szAXYztItyF7PD8vPfEZ0TkHM8+S2WMOQw8ADzs+Uw0sN/z9m0lVj2CuwvuhEXASPE0r0Sk06n2odQJmihUoJgGdBWRNbhbF9tLWecSIElE1uMeRxhvjMnAfeKcLiIbcSeOc73ZoTFmHe6xi1W4xyzeMcasBzoAqzxdQE8Cz5Xy8URg44nB7JP8D/fcxt8Y99Sd4J5LZCuwTkQ2A29TTovfE8sG3GW1X8LduvkB9/jFCd8CbU8MZuNueQR5YtvsWVaqTHp5rFJKqTJpi0IppVSZNFEopZQqkyYKpZRSZdJEoZRSqkyaKJRSSpVJE4VSSqkyaaJQSilVpv8Hk11CIg4zWMAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_roc_curve(fpr, tpr,label = \"SGD\")\n",
    "plot_roc_curve(fpr_forest, tpr_forest, label = \"Random Forest\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如你所见，RandomForestClassifier的 ROC 曲线比SGDClassifier的好得多：它更靠近左上角。所以，它的 ROC AUC 也会更大"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9984428728702259"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "roc_auc_score(y_train_5, y_scores_forest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "precisions, recalls, threshold= precision_recall_curve(y_train_5, y_scores_forest)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.21561531, 0.27633323, 0.33384691, 0.39079299, 0.44420724,\n",
       "       0.49392639, 0.537872  , 0.58108254, 0.62059839, 0.65783044,\n",
       "       0.69319938, 0.7230148 , 0.74968571, 0.77467842, 0.79364608,\n",
       "       0.81018799, 0.82898416, 0.84658188, 0.86105263, 0.87278107,\n",
       "       0.88441624, 0.89404198, 0.90248075, 0.91191261, 0.91814198,\n",
       "       0.92506643, 0.93205793, 0.93697101, 0.94106981, 0.94515539,\n",
       "       0.9495024 , 0.9539547 , 0.95774911, 0.96192989, 0.96508539,\n",
       "       0.96848739, 0.97185813, 0.97418979, 0.97561927, 0.97724151,\n",
       "       0.97887463, 0.98091831, 0.98298639, 0.98348106, 0.98435277,\n",
       "       0.98506547, 0.98598227, 0.98733389, 0.98852254, 0.989498  ,\n",
       "       0.99048626, 0.99065222, 0.99185423, 0.99243243, 0.99389046,\n",
       "       0.99494283, 0.9951122 , 0.99595687, 0.99637763, 0.99657143,\n",
       "       0.99675926, 0.99742087, 0.99762977, 0.99760766, 0.99757105,\n",
       "       0.99753998, 0.99750748, 0.99772899, 0.99769349, 0.99817518,\n",
       "       0.99814274, 0.99810504, 0.9980663 , 0.99831034, 0.99856115,\n",
       "       0.99882041, 0.99909448, 0.999381  , 0.99968153, 0.99966964,\n",
       "       0.9996594 , 0.99964489, 0.99962922, 0.99961656, 0.9996049 ,\n",
       "       0.99958437, 1.        , 1.        , 1.        , 1.        ,\n",
       "       1.        , 1.        , 1.        , 1.        , 1.        ,\n",
       "       1.        , 1.        , 1.        , 1.        , 1.        ])"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precisions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.        , 0.99981553, 0.99926213, 0.99907766, 0.99797085,\n",
       "       0.99760192, 0.99686405, 0.99612618, 0.9948349 , 0.99335916,\n",
       "       0.99280576, 0.99096108, 0.99003874, 0.98874746, 0.98616491,\n",
       "       0.98579598, 0.9845047 , 0.98229109, 0.98081535, 0.97952407,\n",
       "       0.97675706, 0.97435897, 0.9730677 , 0.97011621, 0.96624239,\n",
       "       0.96329091, 0.9616307 , 0.95978602, 0.95738794, 0.95369858,\n",
       "       0.95037816, 0.94779561, 0.94502859, 0.9415237 , 0.93820328,\n",
       "       0.93543627, 0.9300867 , 0.92602841, 0.92270799, 0.91883416,\n",
       "       0.9145914 , 0.91034864, 0.90592142, 0.90057185, 0.89356207,\n",
       "       0.88821251, 0.88230954, 0.87714444, 0.87382402, 0.86902785,\n",
       "       0.86423169, 0.8601734 , 0.85353256, 0.84670725, 0.84025088,\n",
       "       0.83471684, 0.82623132, 0.81793027, 0.81184283, 0.80427965,\n",
       "       0.79431839, 0.78472607, 0.77642501, 0.76923077, 0.7576093 ,\n",
       "       0.74801697, 0.73824018, 0.72938572, 0.71813319, 0.70632725,\n",
       "       0.6939679 , 0.68013282, 0.6664822 , 0.65393839, 0.6401033 ,\n",
       "       0.62479247, 0.61058845, 0.59564656, 0.57904446, 0.55819959,\n",
       "       0.54141302, 0.51927689, 0.49732522, 0.48090758, 0.46670356,\n",
       "       0.44364508, 0.4194798 , 0.39328537, 0.36985796, 0.34421693,\n",
       "       0.30879911, 0.27707065, 0.24737133, 0.21342926, 0.17690463,\n",
       "       0.13576831, 0.09389412, 0.05607821, 0.02434975, 0.        ])"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recalls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在你知道如何训练一个二分类器，选择合适的标准，使用交叉验证去评估你的分类器，选择满足你需要的准确率/召回率折衷方案，和比较不同模型的 ROC 曲线和 ROC AUC 数值。现在让我们检测更多的数字，而不仅仅是一个数字 5。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 多类分类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scikit-Learn 可以探测出你想使用一个二分类器去完成多分类的任务，它会自动地执行 OvA（除了 SVM 分类器，它使用 OvO）。让我们试一下SGDClassifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SGDClassifier(random_state=42)"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sgd_clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['5'], dtype='<U1')"
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sgd_clf.predict([some_digit])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "很容易。上面的代码在训练集上训练了一个SGDClassifier。这个分类器处理原始的目标 class，从 0 到 9（y_train），而不是仅仅探测是否为 5 （y_train_5）。然后它做出一个判断（在这个案例下只有一个正确的数字）。在幕后，Scikit-Learn 实际上训练了 10 个二分类器，每个分类器都产到一张图片的决策数值，选择数值最高的那个类。\n",
    "\n",
    "为了证明这是真实的，你可以调用decision_function()方法。不是返回每个样例的一个数值，而是返回 10 个数值，一个数值对应于一个类。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-30119.38974674, -38865.58222731, -14761.67002307,\n",
       "        -21258.81915546,  -8395.76575415,   4853.46655188,\n",
       "        -23465.69461758, -10519.79517227,  -4220.42364643,\n",
       "         -1233.60193875]])"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "some_digit_scores = sgd_clf.decision_function([some_digit])\n",
    "some_digit_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最高数值是对应于类别 5 ："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.argmax(some_digit_scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['0', '1', '2', '3', '4', '5', '6', '7', '8', '9'], dtype='<U1')"
      ]
     },
     "execution_count": 96,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sgd_clf.classes_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'5'"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sgd_clf.classes_[5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果你想强制 Scikit-Learn 使用 OvO 策略或者 OvA 策略，你可以使用OneVsOneClassifier类或者OneVsRestClassifier类。创建一个样例，传递一个二分类器给它的构造函数。举例子，下面的代码会创建一个多类分类器，使用 OvO 策略，基于SGDClassifier。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['5'], dtype=object)"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.multiclass import OneVsOneClassifier\n",
    "ovo_clf = OneVsOneClassifier(SGDClassifier(random_state = 42))\n",
    "ovo_clf.fit(X_train, y_train)\n",
    "ovo_clf.predict([some_digit])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "45"
      ]
     },
     "execution_count": 99,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(ovo_clf.estimators_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练一个RandomForestClassifier同样简单：\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['5'], dtype=object)"
      ]
     },
     "execution_count": 100,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "forest_clf.fit(X_train, y_train)\n",
    "forest_clf.predict([some_digit])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这次 Scikit-Learn 没有必要去运行 OvO 或者 OvA，因为随机森林分类器能够直接将一个样例分到多个类别。你可以调用predict_proba()，得到样例对应的类别的概率值的列表："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.  , 0.  , 0.01, 0.  , 0.  , 0.86, 0.01, 0.02, 0.01, 0.09]])"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " forest_clf.predict_proba([some_digit])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "你可以看到这个分类器相当确信它的预测：在数组的索引 5 上的 0.8，意味着这个模型以 86% 的概率估算这张图片代表数字 5。它也认为这个图片可能是数字 2 、6或者 8，分别都是 1% 的几率。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在当然你想评估这些分类器。像平常一样，你想使用交叉验证。让我们用cross_val_score()来评估SGDClassifier的精度。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.87525, 0.86735, 0.8664 ])"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cross_val_score(sgd_clf, X_train, y_train, cv=3, scoring=\"accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在所有测试折（test fold）上，它有 86% 的精度。如果你是用一个随机的分类器，你将会得到 10% 的正确率。所以这不是一个坏的分数，但是你可以做的更好。举例子，简单将输入正则化，将会提高精度到 90% 以上。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\dongc\\.conda\\envs\\MachineLearning\\lib\\site-packages\\sklearn\\linear_model\\_stochastic_gradient.py:573: ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.\n",
      "  ConvergenceWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([0.89745, 0.8985 , 0.90015])"
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "scaler = StandardScaler()\n",
    "X_train_scaled = scaler.fit_transform(X_train.astype(np.float64))\n",
    "cross_val_score(sgd_clf, X_train_scaled, y_train, cv=3, scoring=\"accuracy\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 误差分析"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "当然，如果这是一个实际的项目，你会在你的机器学习项目当中，跟随以下步骤（见附录 B）：探索准备数据的候选方案，尝试多种模型，把最好的几个模型列为入围名单，用GridSearchCV调试超参数，尽可能地自动化，像你前面的章节做的那样。在这里，我们假设你已经找到一个不错的模型，你试图找到方法去改善它。一个方式是分析模型产生的误差的类型。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，你可以检查混淆矩阵。你需要使用cross_val_predict()做出预测，然后调用confusion_matrix()函数，像你早前做的那样;"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\dongc\\.conda\\envs\\MachineLearning\\lib\\site-packages\\sklearn\\linear_model\\_stochastic_gradient.py:573: ConvergenceWarning: Maximum number of iteration reached before convergence. Consider increasing max_iter to improve the fit.\n",
      "  ConvergenceWarning)\n"
     ]
    }
   ],
   "source": [
    "y_train_pred = cross_val_predict(sgd_clf, X_train_scaled, y_train, cv=3)\n",
    "conf_mx = confusion_matrix(y_train, y_train_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[5574,    0,   16,    8,    9,   41,   33,    3,  238,    1],\n",
       "       [   1, 6414,   44,   19,    3,   39,    5,    7,  201,    9],\n",
       "       [  26,   28, 5230,   90,   69,   28,   69,   37,  370,   11],\n",
       "       [  21,   21,  104, 5244,    0,  186,   26,   41,  418,   70],\n",
       "       [  10,   16,   39,    9, 5229,    8,   40,   20,  326,  145],\n",
       "       [  23,   20,   32,  159,   49, 4432,   68,   14,  562,   62],\n",
       "       [  27,   18,   46,    3,   41,   84, 5553,    4,  142,    0],\n",
       "       [  16,   11,   51,   27,   49,   13,    4, 5696,  189,  209],\n",
       "       [  16,   67,   39,   98,    3,  113,   28,    8, 5436,   43],\n",
       "       [  22,   20,   29,   55,  133,   33,    1,  176,  366, 5114]],\n",
       "      dtype=int64)"
      ]
     },
     "execution_count": 109,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conf_mx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里是一对数字。使用 Matplotlib 的matshow()函数，将混淆矩阵以图像的方式呈现，将会更加方便。\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPoAAAECCAYAAADXWsr9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAALGElEQVR4nO3dz4vc9R3H8dcr2U1iYvwRWg9upDFabESI0SWoAQ/GQ1tFQXpQMFAve2k1iiDai/+AiB6KsMR6MeghRhAp1oJ66CVkkw2YuKmI2iQaMSHUiBg3m3n3MBNIdrfOd9zvZ78zvp8PCGQn48c3k33m+53Z73zGESEAP29Lmh4AQHmEDiRA6EAChA4kQOhAAoQOJNBY6LZ/a/vftj+x/XRTc1Rl+xrb79uesn3I9vamZ6rC9lLbk7bfbnqWKmxfYXuX7cOdx/r2pmfqxvYTne+Jg7Zfs72i6ZlmayR020sl/VXS7yTdKOkh2zc2MUsPZiQ9GREbJN0m6U8DMLMkbZc01fQQPXhR0jsR8RtJG9Xns9sekfSYpNGIuEnSUkkPNjvVXE0d0TdL+iQiPo2IaUmvS7q/oVkqiYjjEbG/8/tv1f4GHGl2qh9ne62keyTtaHqWKmxfJulOSS9LUkRMR8R/Gx2qmiFJl9gekrRS0pcNzzNHU6GPSDp6wdfH1OfRXMj2OkmbJO1peJRuXpD0lKRWw3NUtV7SCUmvdJ5u7LC9qumhfkxEfCHpOUlHJB2X9E1EvNvsVHM1FbrnuW0grsW1famkNyQ9HhGnm57n/7F9r6SvI2Jf07P0YEjSLZJeiohNkr6T1Nev39i+Uu2z0WslXS1ple2Hm51qrqZCPybpmgu+Xqs+PN2Zzfaw2pHvjIjdTc/TxRZJ99n+XO2nRnfZfrXZkbo6JulYRJw/U9qldvj97G5Jn0XEiYg4K2m3pDsanmmOpkLfK+nXtq+1vUztFy/eamiWSmxb7eeOUxHxfNPzdBMRz0TE2ohYp/bj+15E9N2R5kIR8ZWko7Zv6Ny0VdJHDY5UxRFJt9le2fke2ao+fAFxqIn/aUTM2P6zpH+o/Srl3yLiUBOz9GCLpG2SPrR9oHPbXyLi782N9LP0qKSdnQPAp5IeaXieHxURe2zvkrRf7Z/MTEoab3aquczbVIGfP66MAxIgdCABQgcSIHQgAUIHEmg8dNtjTc/Qi0GbV2LmxdDv8zYeuqS+foDmMWjzSsy8GPp63n4IHUBhRS6YWbNmTYyMVHsz2qlTp7RmzZpK9z148OBCxgJ61r6qtYxSF6tFxJyhi1wCOzIyorfeqv/S9fXr19e+JuYq+c1dSqloVqwot1nM999/X2zt2Th1BxIgdCABQgcSIHQgAUIHEqgU+qDtwQ7gYl1DH9A92AFcoMoRfeD2YAdwsSqhD/Qe7ACqhV5pD3bbY7YnbE+cOnVq4ZMBqE2V0CvtwR4R4xExGhGjVa9dB7A4qoQ+cHuwA7hY1ze1DOge7AAuUOnda50PKeCDCoABxZVxQAKEDiRA6EAChA4kQOhAAkU2h7RdZAOvkp/8umRJmX/zBvHTakvtGTeIj0XJPePOnDlTZN35NofkiA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJEDoQAKVPmTxpyixZXCpLZklaXJyssi6t956a5F1pXLbJ5dat+TfX6mZly1bVmRdqdx2z/PhiA4kQOhAAoQOJEDoQAKEDiRA6EAChA4k0DV029fYft/2lO1DtrcvxmAA6lPlgpkZSU9GxH7bqyXts/3PiPio8GwAatL1iB4RxyNif+f330qakjRSejAA9enpObrtdZI2SdpTZBoARVS+1t32pZLekPR4RJye58/HJI3VOBuAmlQK3faw2pHvjIjd890nIsYljXfuX+YdBgB+kiqvulvSy5KmIuL58iMBqFuV5+hbJG2TdJftA51fvy88F4AadT11j4h/Sar/zeUAFg1XxgEJEDqQAKEDCRA6kAChAwm4xO6Zg3jBzNBQmQ1x9+3bV2RdSdq4cWORdZcvX15k3enp6SLrlnT55ZcXW/v06TkXmC5Yq9VSRMz5KRlHdCABQgcSIHQgAUIHEiB0IAFCBxIgdCABQgcSIHQgAUIHEiB0IAFCBxIgdCABQgcSIHQgAUIHEiB0IAFCBxIgdCABQgcSIHQgAUIHEmC75472p0PXr8Tje96BAweKrHvzzTcXWbfUYyyVe5xXr15dZF1JOnPmTO1rzszMqNVqsd0zkBGhAwkQOpAAoQMJEDqQAKEDCRA6kEDl0G0vtT1p++2SAwGoXy9H9O2SpkoNAqCcSqHbXivpHkk7yo4DoISqR/QXJD0lqVVuFACldA3d9r2Svo6IfV3uN2Z7wvZEbdMBqEWVI/oWSffZ/lzS65Lusv3q7DtFxHhEjEbEaM0zAligrqFHxDMRsTYi1kl6UNJ7EfFw8ckA1IafowMJDPVy54j4QNIHRSYBUAxHdCABQgcSIHQgAUIHEiB0IAF2ge0otUPp8PBwkXUl6ezZs0XWffPNN4us+8ADDxRZV5JarTJXZ1911VVF1pWkkydP1r5mq9VSRLALLJARoQMJEDqQAKEDCRA6kAChAwkQOpAAoQMJEDqQAKEDCRA6kAChAwkQOpAAoQMJEDqQAKEDCRA6kAChAwkQOpAAoQMJEDqQQLFdYEvsqlpi1vNK7QI7iDMvWVLm3/+PP/64yLqSdN111xVZdxB38WUXWCApQgcSIHQgAUIHEiB0IAFCBxIgdCCBSqHbvsL2LtuHbU/Zvr30YADqM1Txfi9Keici/mB7maSVBWcCULOuodu+TNKdkv4oSRExLWm67FgA6lTl1H29pBOSXrE9aXuH7VWF5wJQoyqhD0m6RdJLEbFJ0neSnp59J9tjtidsT9Q8I4AFqhL6MUnHImJP5+tdaod/kYgYj4jRiBitc0AAC9c19Ij4StJR2zd0btoq6aOiUwGoVdVX3R+VtLPzivunkh4pNxKAulUKPSIOSOKUHBhQXBkHJEDoQAKEDiRA6EAChA4kQOhAAsW2e6590cJKbXFccrvnUgZx5qNHjxZZ9/rrry+yrlRmu+4ffvhBrVaL7Z6BjAgdSIDQgQQIHUiA0IEECB1IgNCBBAgdSIDQgQQIHUiA0IEECB1IgNCBBAgdSIDQgQQIHUiA0IEECB1IgNCBBAgdSIDQgQQGahfYUju1SuV2Pi0587lz54qsOzRU9UN2e1NqXqnc39/hw4eLrCtJGzZsqH3NiFBEsAsskBGhAwkQOpAAoQMJEDqQAKEDCRA6kECl0G0/YfuQ7YO2X7O9ovRgAOrTNXTbI5IekzQaETdJWirpwdKDAahP1VP3IUmX2B6StFLSl+VGAlC3rqFHxBeSnpN0RNJxSd9ExLulBwNQnyqn7ldKul/StZKulrTK9sPz3G/M9oTtifrHBLAQVU7d75b0WUSciIizknZLumP2nSJiPCJGI2K07iEBLEyV0I9Ius32StuWtFXSVNmxANSpynP0PZJ2Sdov6cPOfzNeeC4ANar0xuOIeFbSs4VnAVAIV8YBCRA6kAChAwkQOpAAoQMJEDqQQJl9fQtptVrF1m5fC1S/UtsQS9Lw8HCRdWdmZoqsW/KxWL58eZF1N2/eXGRdSdq7d2/ta27btm3e2zmiAwkQOpAAoQMJEDqQAKEDCRA6kAChAwkQOpAAoQMJEDqQAKEDCRA6kAChAwkQOpAAoQMJEDqQAKEDCRA6kAChAwkQOpAAoQMJuMTOnLZPSPpPxbv/QtLJ2ocoZ9DmlZh5MfTLvL+KiF/OvrFI6L2wPRERo40O0YNBm1di5sXQ7/Ny6g4kQOhAAv0Q+njTA/Ro0OaVmHkx9PW8jT9HB1BePxzRARRG6EAChA4kQOhAAoQOJPA/T8m2fOyq/bMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.matshow(conf_mx, cmap=plt.cm.gray)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "让我们关注仅包含误差数据的图像呈现。首先你需要将混淆矩阵的每一个值除以相应类别的图片的总数目。这样子，你可以比较错误率，而不是绝对的错误数（这对大的类别不公平）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [],
   "source": [
    "row_sums = conf_mx.sum(axis=1, keepdims=True)\n",
    "norm_conf_mx = conf_mx / row_sums"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在让我们用 0 来填充对角线。这样子就只保留了被错误分类的数据。让我们画出这个结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPoAAAECCAYAAADXWsr9AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAL2UlEQVR4nO3d34vVdR7H8ddrzoyMo5lJG5LaarTURrQUw1IGXVQX6xZWsBctFGw3QuyWRRC1N/0DEXWxBINuN0VdWBdLLK1BFmwXsmZB5SjYL7PyF6VpNenMvPdijqAz1vke+378ntP7+YBAT8d3byaffs+Z+c5HR4QA/LINNL0AgPIIHUiA0IEECB1IgNCBBAgdSKCx0G3/wfYu27ttP9rUHlXZXmF7i+1x2x/YXt/0TlXYbtl+x/YrTe9She3FtjfZ3tn+WF/f9E6d2H6o/Xvifdsv2B5ueqfZGgnddkvSPyStkXSlpD/bvrKJXbowKenhiPitpOsk/bUPdpak9ZLGm16iC09LejUirpD0O/X47raXSXpA0mhEXCWpJemuZreaq6kr+u8l7Y6IjyLiuKQXJd3e0C6VRMSXEbG9/eOjmvkNuKzZrX6a7eWSbpW0oeldqrC9SNKNkjZKUkQcj4jDjS5VzaCk+bYHJY1I+qLhfeZoKvRlkj475ed71ePRnMr2SknXSNra8CqdPCXpEUnTDe9R1aWSDkp6tv12Y4PtBU0v9VMi4nNJT0jaI+lLSUciYnOzW83VVOg+w2N9cS+u7YWSXpL0YER80/Q+P8b2bZIORMTbTe/ShUFJ10p6JiKukfStpJ7+/I3tCzTzanSVpIslLbB9d7NbzdVU6HslrTjl58vVgy93ZrM9pJnIn4+Il5vep4MbJK21/Ylm3hrdZPu5ZlfqaK+kvRFx8pXSJs2E38tukfRxRByMiBOSXpa0uuGd5mgq9P9J+o3tVbbnaeaTF/9qaJdKbFsz7x3HI+LJpvfpJCIei4jlEbFSMx/f1yOi5640p4qIfZI+s315+6GbJe1ocKUq9ki6zvZI+/fIzerBTyAONvEfjYhJ23+T9B/NfJbynxHxQRO7dOEGSfdIes/2u+3H/h4R/25upV+k+yU9374AfCTp3ob3+UkRsdX2JknbNfOVmXckjTW71Vzm21SBXz7ujAMSIHQgAUIHEiB0IAFCBxJoPHTb65reoRv9tq/EzudCr+/beOiSevoDdAb9tq/EzudCT+/bC6EDKKzIDTO2++4unJm7FzuLiMrPPfn8ftNqtYrM7eZj0e3HeXq6zDfoXXjhhZWe9/3332v+/PldzT506NDZrNRRRMz5wDVyC2wvGhoaKjJ3amqqyNySFi1aVGTu5ORkkbmSdOzYsSJzb7+93DEJGzduLDZ7Nl66AwkQOpAAoQMJEDqQAKEDCVQKvd/OYAdwuo6h9+kZ7ABOUeWK3ndnsAM4XZXQ+/oMdgDV7oyrdAZ7+7t3evrGfiCrKqFXOoM9IsbUPv2yH+91B37Jqrx077sz2AGcruMVvU/PYAdwikrfvdb+Swr4iwqAPsWdcUAChA4kQOhAAoQOJEDoQAKcGdc2MjJSZO53331XZK4knThxosjcUufclTrXTap+uGe3vvhizr1htSlxCOeP/b/jig4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJEDoQAJFjnseGBjQ8PBw7XNLHekrSYcPHy4y97LLLisyV5ImJiaKzP3666+LzF22bFmRuZJ09OjRInPvuOOOInMl6bXXXis2ezau6EAChA4kQOhAAoQOJEDoQAKEDiRA6EACHUO3vcL2Ftvjtj+wvf5cLAagPlVumJmU9HBEbLd9nqS3bb8WETsK7wagJh2v6BHxZURsb//4qKRxSeVucQJQu67eo9teKekaSVuLbAOgiMr3utteKOklSQ9GxDdn+PfrJK1r/7i2BQH8fJVCtz2kmcifj4iXz/SciBiTNCZJrVYratsQwM9W5bPulrRR0nhEPFl+JQB1q/Ie/QZJ90i6yfa77X/+WHgvADXq+NI9Iv4riTfdQB/jzjggAUIHEiB0IAFCBxIgdCCBIqfARoSmp6dLjC7miiuuKDJ3586dReaWtGbNmiJz33zzzSJzJWnevHlF5p5//vlF5krS0qVLa5+5f//+Mz7OFR1IgNCBBAgdSIDQgQQIHUiA0IEECB1IgNCBBAgdSIDQgQQIHUiA0IEECB1IgNCBBAgdSIDQgQQIHUiA0IEECB1IgNCBBAgdSIDQgQSKHfc8OTlZ+9yBgXJ/Lu3bt6/I3FarVWSuJE1NTRWZu3nz5iJzzzvvvCJzJWliYqLI3E8//bTIXElauXJl7TMPHz58xse5ogMJEDqQAKEDCRA6kAChAwkQOpAAoQMJVA7ddsv2O7ZfKbkQgPp1c0VfL2m81CIAyqkUuu3lkm6VtKHsOgBKqHpFf0rSI5Kmy60CoJSOodu+TdKBiHi7w/PW2d5me1tt2wGoRZUr+g2S1tr+RNKLkm6y/dzsJ0XEWESMRsRozTsC+Jk6hh4Rj0XE8ohYKekuSa9HxN3FNwNQG76ODiTQ1fejR8Qbkt4osgmAYriiAwkQOpAAoQMJEDqQAKEDCTgiah/aarVieHi49rkldj1paGioyNzVq1cXmStJ27dvLzL3wIEDReZecsklReZK5U7x/eGHH4rMlaQVK1bUPnP//v06fvy4Zz/OFR1IgNCBBAgdSIDQgQQIHUiA0IEECB1IgNCBBAgdSIDQgQQIHUiA0IEECB1IgNCBBAgdSIDQgQQIHUiA0IEECB1IgNCBBAgdSKDYKbDz58+vfe7k5GTtM09auHBhkblfffVVkbmStHjx4iJzlyxZUmTuhx9+WGSuJNlzDj6tRclTfN96660icyOCU2CBjAgdSIDQgQQIHUiA0IEECB1IgNCBBCqFbnux7U22d9oet3196cUA1Gew4vOelvRqRPzJ9jxJIwV3AlCzjqHbXiTpRkl/kaSIOC7peNm1ANSpykv3SyUdlPSs7Xdsb7C9oPBeAGpUJfRBSddKeiYirpH0raRHZz/J9jrb22xvK3H/PICzVyX0vZL2RsTW9s83aSb800TEWESMRsRoqW8wAHB2OoYeEfskfWb78vZDN0vaUXQrALWq+ln3+yU93/6M+0eS7i23EoC6VQo9It6VNFp2FQClcGcckAChAwkQOpAAoQMJEDqQAKEDCVT9OnpXIkInTpyofe709HTtM08qdTdfiWOvTxoYKPPn9J49e4rMLXnHZKnbrhctWlRkriTdeeedtc/csmXLGR/nig4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJOASp2cODAzE0NBQ7XMvuuii2meedOzYsSJzly5dWmSuJO3atavI3KuvvrrI3N27dxeZK0kTExNF5k5NTRWZK0lLliypfeaRI0c0OTk557hdruhAAoQOJEDoQAKEDiRA6EAChA4kQOhAApVCt/2Q7Q9sv2/7BdvDpRcDUJ+OodteJukBSaMRcZWklqS7Si8GoD5VX7oPSppve1DSiKQvyq0EoG4dQ4+IzyU9IWmPpC8lHYmIzaUXA1CfKi/dL5B0u6RVki6WtMD23Wd43jrb22xvK3H/PICzV+Wl+y2SPo6IgxFxQtLLklbPflJEjEXEaESM2nPuqQfQoCqh75F0ne0RzxR8s6TxsmsBqFOV9+hbJW2StF3Se+1fM1Z4LwA1GqzypIh4XNLjhXcBUAh3xgEJEDqQAKEDCRA6kAChAwkQOpBAseOeh4fr/07WkrfWDg5W+kpj10ZGRorMlaRVq1YVmbtjx44ic48ePVpkriStWbOmyNy1a9cWmStJ9913X5G5EcFxz0BGhA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAoQOJEDoQAKEDiRA6EAChA4kQOhAAkVOgbV9UNKnFZ9+oaRDtS9RTr/tK7HzudAr+/46In41+8EioXfD9raIGG10iS70274SO58Lvb4vL92BBAgdSKAXQh9reoEu9du+EjufCz29b+Pv0QGU1wtXdACFETqQAKEDCRA6kAChAwn8H1Zuw2CSSo53AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 288x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "np.fill_diagonal(norm_conf_mx, 0)\n",
    "plt.matshow(norm_conf_mx, cmap=plt.cm.gray)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "现在你可以清楚看出分类器制造出来的各类误差。记住：行代表实际类别，列代表预测的类别。第 8、9 列相当亮，这告诉你许多图片被误分成数字 8 或者数字 9。相似的，第 8、9 行也相当亮，告诉你数字 8、数字 9 经常被误以为是其他数字。相反，一些行相当黑，比如第一行：这意味着大部分的数字 1 被正确分类（一些被误分类为数字 8 ）。留意到误差图不是严格对称的。举例子，比起将数字 8 误分类为数字 5 的数量，有更多的数字 5 被误分类为数字 8。\n",
    "\n",
    "分析混淆矩阵通常可以给你提供深刻的见解去改善你的分类器。回顾这幅图，看样子你应该努力改善分类器在数字 8 和数字 9 上的表现，和纠正 3/5 的混淆。举例子，你可以尝试去收集更多的数据，或者你可以构造新的、有助于分类器的特征。举例子，写一个算法去数闭合的环（比如，数字 8 有两个环，数字 6 有一个， 5 没有）。又或者你可以预处理图片（比如，使用 Scikit-Learn，Pillow， OpenCV）去构造一个模式，比如闭合的环。\n",
    "\n",
    "分析独特的误差，是获得关于你的分类器是如何工作及其为什么失败的洞见的一个好途径。但是这相对难和耗时。举例子，我们可以画出数字 3 和 5 的例子"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-113-7901b16f7242>, line 7)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;36m  File \u001b[1;32m\"<ipython-input-113-7901b16f7242>\"\u001b[1;36m, line \u001b[1;32m7\u001b[0m\n\u001b[1;33m    plt.subplot(221); plot_digits(X_aa[:25], ../images_per_row=5)\u001b[0m\n\u001b[1;37m                                             ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "cl_a, cl_b = 3, 5\n",
    "X_aa = X_train[(y_train == cl_a) & (y_train_pred == cl_a)]\n",
    "X_ab = X_train[(y_train == cl_a) & (y_train_pred == cl_b)]\n",
    "X_ba = X_train[(y_train == cl_b) & (y_train_pred == cl_a)]\n",
    "X_bb = X_train[(y_train == cl_b) & (y_train_pred == cl_b)]\n",
    "plt.figure(figsize=(8,8))\n",
    "plt.subplot(221); plot_digits(X_aa[:25], ../images_per_row=5)\n",
    "plt.subplot(222); plot_digits(X_ab[:25], ../images_per_row=5)\n",
    "plt.subplot(223); plot_digits(X_ba[:25], ../images_per_row=5)\n",
    "plt.subplot(224); plot_digits(X_bb[:25], ../images_per_row=5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "左边两个5*5的块将数字识别为 3，右边的将数字识别为 5。一些被分类器错误分类的数字（比如左下角和右上角的块）是书写地相当差，甚至让人类分类都会觉得很困难（比如第 8 行第 1 列的数字 5，看起来非常像数字 3 ）。但是，大部分被误分类的数字，在我们看来都是显而易见的错误。很难明白为什么分类器会分错。原因是我们使用的简单的SGDClassifier，这是一个线性模型。它所做的全部工作就是分配一个类权重给每一个像素，然后当它看到一张新的图片，它就将加权的像素强度相加，每个类得到一个新的值。所以，因为 3 和 5 只有一小部分的像素有差异，这个模型很容易混淆它们。\n",
    "\n",
    "3 和 5 之间的主要差异是连接顶部的线和底部的线的细线的位置。如果你画一个 3，连接处稍微向左偏移，分类器很可能将它分类成 5。反之亦然。换一个说法，这个分类器对于图片的位移和旋转相当敏感。所以，减轻 3/5 混淆的一个方法是对图片进行预处理，确保它们都很好地中心化和不过度旋转。这同样很可能帮助减轻其他类型的错误。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 多标签分类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "到目前为止，所有的样例都总是被分配到仅一个类。有些情况下，你也许想让你的分类器给一个样例输出多个类别。比如说，思考一个人脸识别器。如果对于同一张图片，它识别出几个人，它应该做什么？当然它应该给每一个它识别出的人贴上一个标签。比方说，这个分类器被训练成识别三个人脸，Alice，Bob，Charlie；然后当它被输入一张含有 Alice 和 Bob 的图片，它应该输出[1, 0, 1]（意思是：Alice 是，Bob 不是，Charlie 是）。这种输出多个二值标签的分类系统被叫做多标签分类系统。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "目前我们不打算深入脸部识别。我们可以先看一个简单点的例子，仅仅是为了阐明的目的。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KNeighborsClassifier()"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "y_train_int = np.array([int(i) for i in y_train])\n",
    "y_train_large = (y_train_int >= 7)\n",
    "y_train_odd = (y_train_int % 2 == 1)\n",
    "y_multilabel = np.c_[y_train_large, y_train_odd]\n",
    "knn_clf = KNeighborsClassifier()\n",
    "knn_clf.fit(X_train, y_multilabel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这段代码创造了一个y_multilabel数组，里面包含两个目标标签。第一个标签指出这个数字是否为大数字（7，8 或者 9），第二个标签指出这个数字是否是奇数。接下来几行代码会创建一个KNeighborsClassifier样例（它支持多标签分类，但不是所有分类器都可以），然后我们使用多目标数组来训练它。现在你可以生成一个预测，然后它输出两个标签："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[False,  True]])"
      ]
     },
     "execution_count": 120,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_clf.predict([some_digit])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "它工作正确。数字 5 不是大数（False），同时是一个奇数（True）。\n",
    "\n",
    "有许多方法去评估一个多标签分类器，和选择正确的量度标准，这取决于你的项目。举个例子，一个方法是对每个个体标签去量度 F1 值（或者前面讨论过的其他任意的二分类器的量度标准），然后计算平均值。下面的代码计算全部标签的平均 F1 值："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.967980340532212"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_knn_pred = cross_val_predict(knn_clf, X_train, y_train, cv=3)\n",
    "f1_score(y_train, y_train_knn_pred, average=\"macro\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这里假设所有标签有着同等的重要性，但可能不是这样。特别是，如果你的 Alice 的照片比 Bob 或者 Charlie 更多的时候，也许你想让分类器在 Alice 的照片上具有更大的权重。一个简单的选项是：给每一个标签的权重等于它的支持度（比如，那个标签的样例的数目）。为了做到这点，简单地在上面代码中设置average=\"weighted\"。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 多输出分类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们即将讨论的最后一种分类任务被叫做“多输出-多类分类”（或者简称为多输出分类）。它是多标签分类的简单泛化，在这里每一个标签可以是多类别的（比如说，它可以有多于两个可能值）。\n",
    "\n",
    "为了说明这点，我们建立一个系统，它可以去除图片当中的噪音。它将一张混有噪音的图片作为输入，期待它输出一张干净的数字图片，用一个像素强度的数组表示，就像 MNIST 图片那样。注意到这个分类器的输出是多标签的（一个像素一个标签）和每个标签可以有多个值（像素强度取值范围从 0 到 255）。所以它是一个多输出分类系统的例子。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "让我们从 MNIST 的图片创建训练集和测试集开始，然后给图片的像素强度添加噪声，这里是用 NumPy 的randint()函数。目标图像是原始图像。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "noise = rnd.randint(0, 100, (len(X_train), 784))\n",
    "noise = rnd.randint(0, 100, (len(X_test), 784))\n",
    "X_train_mod = X_train + noise\n",
    "X_test_mod = X_test + noise\n",
    "y_train_mod = X_train\n",
    "y_test_mod = X_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "左边的加噪声的输入图片。右边是干净的目标图片。现在我们训练分类器，让它清洁这张图片："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "knn_clf.fit(X_train_mod, y_train_mod)\n",
    "clean_digit = knn_clf.predict([X_test_mod[some_index]])\n",
    "plot_digit(clean_digit)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
